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j  This  report  sunrArizes  orth  research,  over  a  two-year  period,  on  compute 
program  testing,  in  particular  the  development  of  a  strategy  called  the 
Domain  Testing  Strategy,  For  a  large  and  important  class  of  programs,  this 
strategy  enables  the  generation  of  test  data  vhich  can  test,  in  principle, 
for  all  errors  in  the  control  flow  of  a  program.  Among  the  constraints  for 
practical  application  of  the  strategy  is  that  the  predicates  that  affect  the 
control  flow  are  linear  in  the  input  variables.  Hie  extension  of  the{strate( 
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I.  INTRODUCTION 

Most  of  Che  emphasis  In  our  work  during  the  two  years  has  been  on  reliable 
software  in  general,  and  program  testing  in  particular.  We  have  developed  a 
testing  strategy  called  Domain  Testing  Strategy  which  is  very  promising  for  a 
large  class  of  data  processing  programs.  Our  efforts  have  been  devoted  to 
both  theoretical  and  practical  aspects  of  the  strategy.  At  Che  theoretical 
level,  we  have  delineated  Che  precise  conditions  under  which  the  strategy  Is 
guaranteed  to  detect  certain  classes  of  errors,  specified  the  sensitivity  of 
the  strategy  to  certain  error  parameters  in  the  choice  of  test  data,  and 
obtained  preliminary  results  on  extending  the  strategy  to  large  programs  con¬ 
structed  out  of  modules  of  snail  programs.  At  the  practical  level,  we  have 
been  developing  a  prototype  test  system  based  on  the  strategy. 

In  this  Report,  we  shall  take  the  approach  of  presenting  the  main  results 
obtained  in  brief,  intuitively  meaningful  terms  and  leave  the  technical 
details  to  several  appendices.  Some  of  these  appendices  are  copies  of  papers 
published  in  the  open  literature,  and  some  are  technical  reports. 

II.  THEORETICAL  ISSUES 

11 .  1.  Domain  Testing  Strategy 

Computer  programs  contain  two  types  of  errors  which  have  been  identified  as 
computation  errors  and  domain  errors.  A  domain  error  occurs  when  a  specific 
input  follows  the  wrong  path  due  to  an  error  in  the  control  flow  of  the  pro¬ 
gram.  A  path  contains  a  computation  error  when  a  specific  input  follows  the 
correct  path,  but  an  error  in  sone  assignment  statement  causes  the  wrong  func¬ 
tion  to  be  computed  for  one  or  more  of  the  output  variables.  A  testing  stra¬ 
tegy  has  been  designed  to  detect  domain  errors,  and  the  conditions  under 
which  this  strategy  is  reliable  are  given  and  charac terlzed .  A  by-product 
of  this  domain  strategy  is  a  partial  ability  to  detect  computation  errors. 

It  is  the  objective  of  this  study  to  provide  an  analytical  foundation  upon 
whlch  to  base  practical  testing  implementations. 

There  are  limitations  inherent  to  anv  testing  strategy,  and  these  also  constrain 
the  proposed  domain  strategy.  One  such  limitation  might  be  termed  coinciden¬ 
tal  correctness,  which  occurs  when  a  specific  test  point  follows  an  incorrect 
path,  and  yet  the  output  variables  coincidentally  are  the  same  as  if  that 
test  point  were  to  follow  the  correct  path.  This  test  point  would  then  he 
of  no  assistance  in  the  detection  of  the  domain  error  which  caused  the  control 
flow  change.  No  test  generation  strategy  can  circvmvent  this  problem. 

Another  inherent  testing  limitation  has  been  previously  identilied  as  a  missing 
path  error,  in  which  a  required  predicate  does  not  appear  in  the  given  pro¬ 
gram  to  be  tested.  Especially  if  th.is  predicate  were  an  equality,  no  testing 
strategy  could  systematically  determine  that  such  a  predicate  should  be  present. 

The  control  flow  statements  In  a  computer  program  partition  the  input  space 
Into  a  set  of  mutually  exclusive  doma ins .  each  of  which  corresponds  to  a 
particular  program  path  and  consists  of  input  data  points  which  cause  that  path 
to  be  executed.  The  testing  strategy  generates  test  points  to  examine  the 
boundaries  of  a  domain  to  detect  whether  a  domain  error  has  occurred,  as  either 
one  or  more  of  these  boundaries  will  have  shifted  or  else  the  corresponding 
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relational  operator  has  changed.  If  test  ooints  can  be  chosen  within 

boundary,  the  strategy  is  shown  to  be  reliable  in  detecting  domain 

magnitude  greater  than  e  subject  to  the  following  assumptions: 

coincidental  correctness  does  not  occur; 

missing  path  errors  do  not  occur; 

predicates  are  linear  in  the  input  variables; 

the  input  space  is  continuous. 


Assumptions  (1)  and  (2)  have  been  shown  to  be  inherent  to  the  testing  process, 
and  cannot  be  entirely  eliminated.  However,  recognition  of  these  potential 
problems  can  lead  to  improved  testing  techniques.  The  domain  testing  method 
has  been  shown  to  be  applicable  for  nonlinear  boundaries  but  the  number  of 
required  test  points  may  become  inordinate  and  there  are  complex  problems 
associated  with  processing  nonlinear  boundaries  in  higher  dimensions.  The 
continuous  input  space  assumption  is  not  really  a  limitation  of  the  proposed 
testing  method,  but  allows  the  parameter  c  to  be  chosen  arbitrarily  small. 

An  error  analysis  for  discrete  spaces  is  available  and  the  testing  strategy 
has  been  Droved  viable  as  long  as  the  size  of  the  domain  is  not  comparable 
to  the  discrete  resolution  of  the  space. 


Next  let  us  consider  two  further  assumptions: 

(5)  predicates  are  simple;  and 

(6)  adjacent  domains  conpute  different  functions. 


If  assumptions  (5)  and  (6)  are  Imposed,  the  testing  strategy  is  considerably 
simplified,  as  no  more  than  one  domain  need  be  examined  at  one  time  in  order 
to  select  test  points.  Moreover,  the  number  of  test  points  required  to  test 
each  domain  grows  linearly  with  both  the  dimensionality  of  the  Input  space 
and  the  number  of  predicates  along  the  path  being  tested. 

The  only  completely  effective  testing  strategy  is  an  exhaustive  test  which 
is  totally  impractical.  The  domain  testing  strategy  offers  a  substantial 
reduction  in  the  high  cost  of  computer  program  testing,  and  yet  can  reliably 
detect  a  major  class  of  errors  which  have  been  characterized.  In  addition, 
other  types  of  errors  can  be  detected,  such  as  computation  errors  and  missing 
path  errors,  but  this  detection  cannot  be  guaranteed. 


The  domain  strategy  is  currently  being  implemented,  and  will  be  utilized  as 
an  experimental  facility  for  subsequent  research.  A  most  important  contri¬ 
bution  would  be  to  indicate  both  programing  language  constructs  and  program¬ 
ming  techniques  which  are  easier  to  test,  and  thus  produce  more  reliable  soft¬ 
ware  . 


The  most  comprehensive  presentation  of  results  to  date  1®  available  in  (1),  [3], 
and  (6J.  [1]  and  [3]  are  attached  as  appendices  to  this  report. 

II .  2.  Error  Analysis 

The  objective  is  to  provide  an  error  analysis  of  the  domain  testing  strategy. 

It  has  been  shown  that  some  border  shifts  will  escape  detection  by  the  strategy; 
this  occurs  because  either  the  test  points  are  not  selected  appropriately,  or 
else  the  border  shift  Is  too  close  to  the  given  border  to  be  detected  by  the 
selected  test  point.  An  error  analysis  will  Indicate  the  best  locations  for  the 
test  points. 
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The  strategy  was  developed  for  continuous  spaces,  but  computer  representation 
may  have  to  be  examined  as  a  discrete  space  in  order  to  assure  us  that  roundoff 
will  not  introduce  unacceptable  errors.  It  has  been  shown  that  there  are  some 
domains  in  a  discrete  space  which  cannot  be  tested  by  the  strategy,  but  these 
are  pathological  cases  where  one  of  the  domain  dimensions  is  on  the  order  of  the 
lattice  resolution.  Moreover,  a  simple  computation  can  be  made  to  indicate  when 
this  condition  exists  for  a  given  domain. 

An  error  analysis  of  domain  borders  is  needed  to  resolve  the  following  questions 
i)  How  small  should  c  be  chosen  In  selecting  an  OFF  test  point  for  linear 
borders,  and  where  are  optimal  locations  for  the  test  points? 
ii)  We  required  the  OFF  test  point  for  a  given  border  to  satisfy  all 

Inequality  borders  except  that  being  tested;  how  do  potential  errors 
in  other  borders  of  the  domain  affect  this  requirement? 

Ill)  What  are  the  difficulties  in  applying  domain  testing  In  a  discrete 

space  or  in  a  space  in  which  numerical  values  can  only  be  represented 
with  finite  resolution,  and  can  these  difficulties  be  circumvented 
by  taking  reasonable  precautions  with  the  method? 

These  and  other  error  analysis  problems  are  dealt  with  in  detail  in  reference 
[2]  (see  list  of  publications  from  existing  research.  Section  V.)  Chapter  6 
of  Appendix  A  gives  a  capsule  summary  of  the  error  analysis  results. 

I I .  3.  Summary  of  Basic  Results 

The  basic  goal  of  this  research  is  to  replace  the  Intuitive  principles  behind 
Current  testing  procedures  by  a  methodology  based  on  a  formal  treatment  of  the 
program  testing  problem.  By  formulating  the  problem  in  basic  geometric  and 
algebraic  terms,  we  have  been  able  to  develop  an  effective  testing  methodology 
whose  capabilities  can  be  precisely  defined.  In  addition,  since  program  testing 
cannot  be  completely  effective,  we  have  identified  the  limitations  of  the 
strategy.  In  several  cases  these  limitations  have  proven  to  be  theoretical 
problems  Inherent  to  the  general  program  testing  process. 

The  main  contribution  of  this  research  is  the  development  of  the  domain  testing 
strategy.  Under  certain  well-defined  conditions  the  methodology  is  guaranteed 
to  detect  domain  errors  in  linear  borders  greater  than  some  small  magnitude  c. 
Furthermore,  the  cost,  as  measured  by  the  number  of  required  test  points,  is 
reasonable  and  grows  only  linearly  with  both  the  dimensionality  of  the  innut 
space  domain  and  the  number  of  path  predicates.  Domain  testing  also  detects 
transformation  errors  and  missing  path  errors  in  manv  cases,  but  the  detection 
of  these  two  classes  of  errors  cannot  be  guaranteed. 

Domain  testing  has  also  been  extended  to  classes  of  nonlinear  borders,  and  we 
have  shown  that  the  methodology  generalizes  to  anv  class  of  functions  which 
can  be  described  by  a  finite  number  of  parameters.  Unfortunately,  nonlinear 
predicates  pose  problems  of  extra  processing  which  probably  preclude  testing 
except  for  restricted  cases.  For  example.  Just  finding  intersection  points 
of  a  set  of  linear  and  nonlinear  borders  can  require  an  inordinate  amount  of 
process  ing. 

Coincidental  correctness  is  a  theoretical  limitation  Inherent  to  the  program 
testing  process,  and  we  have  argued  that  it  prevents  any  reasonable  finite 
testing  procedure  from  being  completely  reliable.  In  particular,  the  nossi- 
bllity  of  coincidental  correctness  means  that  an  exhaustive  test  of  all  points 
in  an  input  domain  is  theoretically  required  to  preclude  the  existence  of 
computation  errors  on  a  path.  Within  the  class  of  all  computable  functions 
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there  exist  functions  which  coincide  at  an  arbitrarily  large  number  of  points, 
but  if  there  is  sufficient  resolution  in  the  output  space,  coincidental  correct¬ 
ness  should  be  a  rare  occurrence  for  functions  commonly  encountered  in  data 
processing  problems. 

The  class  of  missing  path  errors,  particularlv  those  of  reduced  dimensionality, 
has  proven  to  be  another  theoretical  limitation  to  the  reliabilitv  of  any  finite 
testing  strategy.  Although  our  methodology  cannot  be  guaranteed  to  detect  all 
instances  of  this  type  or  error,  it  can  be  extended  to  detect  some  well-defined 
subclasses  of  missing  path  errors.  Unfortunately,  the  extra  cost  of  this 
modification  may  be  unacceptably  high.  Our  analysis  of  missing  path  errors 
has  shown  that  the  cause  of  the  difficulty  is  that  the  program  does  not  contain 
any  indication  of  the  possible  existence  of  a  missing  path  error.  Therefore, 
without  additional  information,  a  reasonable  testing  strategy  for  this  class 
of  errors  cannot  be  formulated. 

The  domain  testing  strategy  requires  a  reasonable  number  of  test  points  for 
a  single  path,  but  the  total  cost  may  be  unacceptable  for  a  large  program 
containing  an  excessive  number  of  paths.  In  particular,  this  nav  occur  for  large 
programs  with  complicated  control  structures  containing  many  iteration  loops. 
Additional  research  is  needed  to  substantially  reduce  the  number  of  potential 
paths . 

1 1 .  4.  Extension  of  Strategy  to  Modules 

The  major  drawback  of  the  Domain  Testing  Strategy  in  its  current  state  of 
development  is  that  it  requires  testing  all  of  the  possible  paths  thru  the 
program  being  tested.  As  programs  increase  in  complexity  the  number  of 
possible  paths  increases  dramatically.  This  presents  a  severe  practical 
constraint  on  the  ability  to  test  programs  of  reasonable  size.  (It  should 
be  noted  that  this  problem  is  inherent  in  all  path  oriented  testing  strategies, 
and  not  Just  the  Domain  Testing  Strategy.) 

One  possible  approach  to  reducing  this  testing  problem  is  motivated  by  con¬ 
sidering  the  problem  of  program  development.  Here,  too  we  nay  be  faced  with 
a  large,  complicated,  unmanageable  task  when  the  problem  is  considered  in  its 
entiretv.  The  suggested  methodology  in  this  case  is  to  consider  the  overall 
problem  as  a  set  of  related  units,  and  to  develop  the  details  of  the  solution 
around  this  modular  structure.  In  a  similar  manner,  a  testing  strategy  could 
make  use  of  the  notion  of  modular  structure. 


If  different  segments  of  the  program  are  developed  and  tested  independently, 
and  later  Integrated  to  form  the  final  version  of  the  program  (a  kind  of 
"bottom  up"  approach),  it  would  be  nice  if  the  validation  information  obtained 
through  these  "unit  tests"  can  be  used  to  substantially  reduce  the  amount 
of  testing  Chat  needs  to  be  done  when  considering  the  entire  program  at 
integration  time.  Thus,  the  primary  Justification  for  the  development  of 
a  method  of  integrating  Independently  tested  program  modules  into  a  single 
tested  program  is  to  cut  down  the  total  number  of  paths  that  need  be  ‘ested, 
and  to  keep  the  total  number  of  oaths  reasonable  as  programs  inen  in 
complexity.  A  secondary  Justification  for  an  independent  testing  tegv  is 
to  make  the  testing  procedure  for  a  program  conform  to  the  wav  pro-'  s  are 
developed.  By  modularizing  the  testing  procedure  the  overall  task  testing 
a  large  program  becomes  more  manageable. 
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We  define  a  module  as  a  block  of  single  entry  single  exit  code,  which  can 
contain  an  arbitrary  amount  of  computation  and  Internal  control  structure. 

Using  this  characterization  of  module,  we  address  the  problem  of  integrating 
Independently  tested  modules  into  the  testing  of  a  program  which  incorporates 
the  Domain  Testing  Strategy.  It  will  be  useful  to  consider  the  following  two 
components  of  this  problem  separately. 

1.  Given  an  untested  program  that  uses  one  or  more  modules  that  are  knovn  to 
be  correct,  how  can  the  Domain  Testing  Strategy  be  applied  to  the  program  in 
order  to  take  maximum  advantage  of  the  correctness  of  these  nodules. 

2.  Given  an  untested  module,  how  much  testing  need  be  done  on  the  module  in 
order  to  incorporate  it  into  the  Domain  Testing  of  a  comnlete  program  with  a 
mlnumum  of  testing  overhead.  Specifically,  is  it  enough  to  Domain  Test  the 
modules  which  are  to  be  incorporated  into  the  testing  strategy  developed  in 
the  solution  of  the  first  problem. 

In  examining  the  first  problem  it  is  clear  that  the  ideal  solution  would  allow 
the  testing  of  the  program  to  be  performed  without  having  to  consider  the 
complexity  of  the  correct  modules.  If  this  can  be  accomplished ,  then  when 
testing  the  program  the  correct  module  can  be  treated  as  a  form  of  assignment 
statement.  The  actual  control  structure  of  the  module  wouldn't  need  to  be 
considered.  This  type  of  solution  is  intuitively  appealing  because  it  wouldn't 
require  additional  testing  of  a  block  of  code  that  is  already  known  to  be 
correct.  However,  such  a  solution  would  only  be  acceptable  if  it  didn't  result 
in  the  loss  of  a  large  amount  of  the  testing  confidence  that  would  have 
been  obtained  if  each  path  through  the  program  had  been  tested  using  the 
Domain  Testing  Strategy. 

Therefore,  we  shall  assume  that  the  above  technique  of  integrating  independently 
tested  modules  will  be  used,  and  to  analyze  the  types  of  errors  which  this 
technique  will  allow  to  go  undetected.  Instead  of  looking  at  all  types  of 
undetectable  errors,  however,  we  will  only  be  concerned  with  the  types  of  errors 
that  a  complete  Domain  Testing  of  each  path  would  have  detected,  but  the 
Integrated  approach  would  miss.  The  types  of  errors  that  complete  Domain 
Testing  wouldn't  detect  will  be  assumed  to  be  undetected  using  the  integrated 
approach  also.  We  therefore  next  examine  the  types  of  errors  that  can  occur 
in  a  program  which  the  Domain  Testing  Strategy  will  detect. 

The  purpose  of  the  Domain  Testing  Strategy  is  to  detect  errors  in  the  control 
structure  of  a  program.  There  are  two  ways  that  an  error  can  occur  in  a  pro¬ 
gram’s  control  structure:  the  actual  predicate  could  be  incorrect,  or  a  compu¬ 
tation  that  occurs  along  some  path  In  the  program  and  is  then  used  in  a 
predicate  could  be  incorrect .Given  these  two  types  of  errors  the  following 
five  cases  can  occur  when  integrating  a  correct  module  into  a  program  being 
tested.  (In  this  context,  wc  mean  by  "program"  the  integrated  code  excluding 
the  correct  nodule.) 

1.  A  predicate  in  the  program  could  be  Incorrect. 

2.  A  computation  in  the  program  could  be  incorrect,  and  that  computation 
is  used  in  a  predicate  later  in  the  program. 

3.  A  computation  in  the  program  is  incorrect,  and  the  computation  Is 
used  in  a  predicate  of  the  correct  module,  but  isn't  used  later  in  a 
program  predicate. 

4.  A  computation  in  the  program  is  incorrect,  and  the  result  of  the 
computation  is  not  used  in  a  predicate  in  either  the  module  or  the 
program,  but  is  used  in  a  computation  in  the  module. 
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5.  A  computation  in  the  program  Is  incorrect,  and  the  result  of  the 

computation  is  used  in  a  later  program  computation  but  is  not  used  at 
all  in  the  module. 

In  the  first  and  second  cases  Domain  Testing  will  detect  these  errors  as  a 
shift  in  the  input  domain  of  the  program.  Since  neither  type  of  error  would 
affect  the  correct  module,  using  the  integrated  testing  approach  would  also 
detect  these  types  of  errors.  Case  5  also  does  not  affect  the  correct  module, 
so  the  error  would  be  detected  to  the  extent  that  Domain  Testing  is  lucky  enough 
to  catch  computation  errors.  However,  since  not  as  many  points  are  being  tested 
with  the  integrated  strategy  we  would  expect  some  degradation  in  testing 
conf idence . 

In  the  third  case  there  is  a  possibility  that  the  error  would  go  undetected 
using  the  integrated  approach,  yet  would  have  been  detected  if  all  paths  had 
been  Domain  Tested.  The  problem  occurs  because  the  predicate  in  the  correct 
module  that  would  have  detected  the  error  mav  not  be  executed  since  only  one  of 
an  arbitrarily  large  number  of  paths  through  the  correct  module  will  be  exe¬ 
cuted.  The  fourth  case  might  also  go  undetected  with  the  integrated  approach. 
However,  even  if  the  computation  in  the  correct  module  that  uses  the  incorrect 
computation  from  the  program  lies  on  the  path  that  is  taken  through  the  correct 
module,  the  error  might  still  go  undetected.  This  is  a  case  in  which  both  the 
Integrated  approach  and  Domain  Testing  might  miss  the  error,  but  once  again 
it  is  important  to  note  that,  using  the  integrated  approach,  it  appears  that 
there  is  even  less  chance  of  catching  the  error  than  with  Domain  Testing. 

The  types  of  errors  that  might  go  undetected  using  the  integrated  approach, 
but  would  be  detected  by  Domain  Testing  each  nath,  are  sufficiently  serious 
to  require  some  modification  to  the  method  of  testing  programs  that  contain 
correct  nodules.  The  corrson  feature  of  both  tvpes  of  errors  (cases  3  and  4 
above)  is  that  there  is  a  computation  error  which  is  actually  in  the  program, 
but  only  shows  up  because  of  its  effect  on  the  correct  nodule.  One  method 
of  avoiding  this  problem  would  be  to  require  that  in  testing  the  program  both 
the  output  from  the  progran,  and  the  values  that  are  generated  in  the  program 
and  used  by  the  correct  module,  be  validated.  This  in  effect  corresponds  to 
validating  the  inputs  to  the  correct  module,  and  could  be  accomplished  in  two 
ways.  First,  an  additional  burden  could  be  placed  on  the  oracle  while  testing 
the  program,  that  burden  being  the  validation  of  the  inputs  to  the  correct 
module.  A  second,  more  appealing  approach  would  be  to  treat  the  section  of 
code  preceding  the  correct  module  as  a  separate  module  which  itself  would  be 
tested  independently.  While  this  section  was  being  tested  independently  its 
outputs  would  be  validated,  thereby  validating  the  inputs  to  the  correct  module. 

For  this  second  approach  to  work  it  must  be  shown  that  the  independent  testing 
of  this  section  is  sufficient  to  detect  the  types  of  computation  errors  that 
might  cause  integration  test  problens.  This  leads  to  consideration  of  the 
second  problem,  identified  earlier,  of  developing  a  method  of  separately  testing 
progran  modules  to  be  integrated  into  the  testing  of  the  conplete  program. 

I’p  to  this  point  it  has  been  assumed  that  the  module  that  is  being  integrated 
into  the  program  being  tested  is  completely  correct.  In  general  this  will  not 
be  the  case,  especially  if  the  module  has  been  validated  through  testing, 
since  no  practical  testing  strategy  can  guarantee  the  correctness  of  a  progran 
of  module.  Ideally  we  would  like  to  be  able  to  use  the  Domain  Testing  Strategy 
on  the  module,  and  then  use  the  method  described  previously  for  testing  the 
(integrated)  program  without  having  to  retest  the  paths  through  the  module. 
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Since  Che  ultimate  goal  is  to  Domain  Test  the  program.  It  Is  necessary  for  the 
Independent  testing  of  the  module  to  identify  all  errors  in  a  predicate,  as 
well  as  all  errors  in  computations  that  will  be  used  later  in  a  predicate  in 
either  the  module  being  tested  or  in  the  integrated  program.  Bv  Domain  Testing 
the  module, the  errors  in  the  predicates  in  the  module  as  well  as  computation 
errors  which  affect  predicates  in  the  module,  will  be  detected  as  thev  will 
cause  a  shift  in  a  border  in  the  domain  of  the  module.  However,  in  general 
many  types  of  errors  in  the  computations  performed  in  the  module  might  go 
undetected  if  they  aren't  subsequently  used  in  a  predicate  in  the  module. 

If  the  programs  under  consideration  contained  only  linear  predicates  (when 
viewed  with  respect  to  the  entire  program)  then  a  good  deal  of  the  problem  with 
computation  errors  can  be  eliminated  by  Domain  Testing.  This  is  due  to  the 
fact  that  if  only  linear  predicates  are  being  tested  then  all  linear  computa¬ 
tions  in  the  module  can  be  validated.  If  the  module  is  Domain  Tested  then 
for  each  subdomain  in  the  module  a  sufficient  number  of  test  points  are  gene¬ 
rated  by  the  testing  strategy  to  span  the  space  of  the  subdomain.  Therefore, 
if  the  linear  computations  are  shown  to  be  correct  on  these  test  points  then 
these  computations  can  be  assumed  to  be  correct  for  anv  point  in  the  subdomain. 
This  means  that  there  can  be  no  computation  errors  in  a  Domain  Tested  module 
that  can  affect  a  predicate  in  the  integrated  program,  so  that  an  integration 
test  which  ignored  the  paths  in  the  Domain  Tested  module  will  be  Just  as  ef¬ 
fective  as  a  test  which  didn't  ignore  them,  with  respect  to  these  kind  of  errors. 

There  can,  however,  still  be  errors  in  nonlinear  computations  of  a  Domain  Tested 
module  which  affect  later  computations  in  the  program.  If  the  integration  test 
ignored  the  paths  in  the  Domain  Tested  module,  then  it  is  certainly  possible  for 
the  error  not  to  show  up,  for  those  paths  exercised  in  the  integration  test 
nay  not  contain  the  error  and  the  set  of  points  chosen  in  the  unit  test  may 
not  have  been  sufficient  to  detect  the  error.  Even  if  the  integration  test 
considered  all  of  the  paths  in  the  module,  this  kind  of  error  night  still  go 
undetected  (depending  on  the  kind  of  nonlinearity  in  the  '-imputation  and  the 
number  of  points  tested),  though  the  increased  number  of  test  points  reduce  the 
chances  that  this  will  happen. 

If  there  were  linear  predicates  in  each  nodule  (with  respect  to  the  module's 
inputs)  but  these  predicates  weren't  necessarily  linear  when  viewed  with 
respect  to  the  entire  program,  then  it  is  no  longer  the  case  that  there  can 
he  no  computation  errors  in  the  earlier  nodule  which  affect  a  predicate  in 
the  later  module.  This  is  because  there  must  be  a  nonlinear  computation  in 
the  earlier  module,  and  If  all  of  the  predicates  in  that  module  were  linear, 
we  may  not  have  tested  enough  points  to  guarantee  the  correctness  of  the  non¬ 
linear  computation.  Vhen  Integrating  the  Domain  Tested  module  and  ignoring 
fts  paths  (I.e,  treating  It  as  an  assignment  statement),  we  may  find  that 
the  resulting  output  for  the  test  points  chosen  Is  correct  because  not  enough 
points  were  chosen  to  make  the  nonlinear  computation  error  affect  the  predicate 
(which  looked  linear  vhen  it  was  tested).  However,  if  we  Domain  Tested  the 
complete  program  (ignoring  modularity),  this  predicate  would  have  shown  up 
as  nonlinear  and  enough  points  would  have  been  required  by  the  testing  strategy 
to  detect  this  error. 

It  therefore  makes  a  difference,  when  considering  independent  module  testing 
using  the  Domain  Testing  Strategy,  as  to  the  kind  of  linearity  restrictions 
placec  on  the  program  being  tested.  Of  course,  the  severity  of  this  problem 
with  respect  to  Domain  Testing  is  not  clear,  since  there  are  serious  practical 
problems  associated  with  the  strategy  when  nonlinear  borders  are  involved. 
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One  additional  bit  of  overhead  that  would  arise  in  the  independent  testing  of 
a  aodule  would  be  in  identifying  the  input  space  of  the  module.  Since  the 
module  lies  within  the  program,  its  input  domain  could  consist  of  both  input 
variables  and  program  variables.  The  oracle  would  have  to  be  sufficiently 
knowledgable  to  be  able  to  determine  the  correctness  of  the  results  of  the 
aodule  for  any  values  in  the  module's  (as  opposed  to  the  program  s)  input 
space.  But  this  requirement  doesn't  appear  unreasonable  in  view  of  its  con¬ 
sistency  with  current  views  on  program  development. 

In  conclusion  it  appears  that  a  method  of  testing  modules  independently  can 
prove  to  be  effective  with  little  loss  of  confidence  in  the  testing  procedure. 
The  aajor  limitations  are  in  the  restriction  of  the  Domain  Testing  procedure  to 
onlv  linear  borders,  and  in  the  additional  burden  that  is  placed  on  the  oracle 
that  determines  the  correctness  of  the  testing  results. 

III.  IMPLEMENTATION  OF  A  PROTOTYPE  TESTING  SYSTEM 

We  are  currently  Implementing  a  prototype  system  as  an  experimental  facility. 
This  will  continue  to  be  a  major  focus  of  research  during  the  next  year. 

The  system  is  composed  of  five  phases,  the  first  three  are  written  in  FL/I, 
and  the  other  two  in  Fortran.  The  system  accepts  a  user  program  written  in 
a  subset  of  PL/C  and  performs  the  Domain  Testing  Strategy  on  this  program. 

There  are  several  restrictions  on  the  user  PL/C  program  for  theoretical  inple- 
mentational  reasons.  Work  Is  being  done  to  relax  these  restrictions  by 
expanding  the  system  and  studying  the  theoretical  problems.  The  current  re¬ 
strictions  are:  no  arrays,  no  subroutines,  no  alphanumeric  variables,  single 
entrv/single  exit  blocks,  and  only  two  Input  variables. 

Once  the  user  program  is  submitted  to  the  system,  Phase  One  parses  the  user 
program  and  loads  the  parsed  program  into  a  large  array,  with,  each  record 
containing  approximately  one  statement.  Phase  One  recognizes  the  type  of  state 
nent,  and  through  recursion  it  realizes  the  range  of  each  type  of  control  struc 
ture.  Pointers  in  the  table  are  set  to  indicate  the  end  of  the  control  struc¬ 
ture.  There  are  three  types  of  control  structures: 

(1)  DO  loops. 

(2)  IF  THEN  ELSE  statements. 

(3)  IF  THEN  statement. 

Once  Phase  One  is  finished,  Phase  Two  steps  through  the  parsed  program  checking 
the  type  of  statement.  If  the  statement  contains  some  kind  of  arithmetic 
computat ion ,  this  phase  then  checks  the  linearity  of  the  statement  in  terms 
of  the  variables.  If  the  statement  is  linear  then  the  statement  is  put  in 
a  standard  form  for  further  processing.  If  the  statement  is  non-linear,  the 
statement  is  flagged. 

Phase  Three  takes  a  path  through  the  program  specified  by  the  user,  and  sym¬ 
bolically  executes  the  path  to  produce  a  set  of  predicates  that  describe  the 
path.  The  predicates  are  in  terms  of  the  input  variables  as  the  result  of  the 
symbolic  execution,  and  the  predicates  together  form  the  domain  of  the  path 
in  the  input  space. 

Phase  Four  takes  the  set  of  predicates  generated  for  the  path,  and  performs 
a  Gaussian  elimination  to  get  a  description  of  the  actual  domain  by  way  of 
the  constraints. 
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Phase  Five  Cakes  the  domain  generated  In  phase  four  and  plots  the  path  domain 
when  there  are  only  two  Input  variables.  Also  test  points  are  generated  based 
on  the  border  of  Che  domain.  For  each  border  there  are  two  ON  points  and  one 
OFF  point  generated.  These  test  points  can  then  be  used  as  inputs  to  test 
the  output  and  detect  shifts  in  the  domain  borders. 

Future  extensions  of  the  system  deal  with  restrictions  on  the  user  language, 
and  the  scope  of  the  system.  These  extensions  are: 

(1)  Allow  subroutine. 

(2)  Allow  more  than  two  input  variables. 

(3)  Allow  the  use  of  arrays  and  non-linear  expressions. 

(4)  Studv  compound  predicates. 

(5)  Allow  alphabetic  and  alphanumeric  variables  and  study  their  effects 
on  input  domains. 

(6)  Change  the  user  and  system  language  to  allow  for  portability  and 
more  flexible  use. 


IV.  OTHER  TECHNICAL  ACTIVITIES 

We  have  regarded  participation  in  national  and  International  professional  ac¬ 
tivities  on  computer  program  testing  as  an  important  component  of  our  research 
under  this  Grant.  The  following  activities  in  this  connection  are  worth 
nent toning. 

(a)  Professor  Chandrasekaran,  one  of  the  Principal  Investigators,  was 

an  organizer,  panelist  and  session  chairman  at  the  Workshop  on  Soft- 
Ware  Testing  and  Test  Documentation,  held  during  December  1978  in 
Ft.  Lauderdale,  Fla.,  under  the  auspices  of  the  IEEE. 

(b)  Professor  Chandrasekarar.  has  undertaken  to  edit  a  Special  Issue  of  the 
IEEE  Transactions  on  Software  Engineering  devoted  to  papers  on 
program  testing.  Most  of  the  editorial  work  was  done  under  the 
auspices  of  the  Grant.  The  issue  itself  will  appear  either  in  late 
1979  or  ear  1 v  1980. 
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EXTENDED  ABSTRACT 


Computer  programs  contain  two  types  of  errors  which  have  been  identified  as 
computation  errors  and  domain  errors.  A  domain  error  occurs  when  a  specific  input 
follows  the  wrong  path  due  to  an  error  in  the  control  flow  of  the  program.  A  path 
contains  a  coraputat ion  error  when  a  specific  input  follows  the  correct  path,  but  an 
error  in  some  assignment  statement  causes  the  wrong  function  to  be  computed  for  one 
or  more  of  the  output  variables.  A  testing  strategy  has  been  designed  to  detect 
domain  errors,  and  the  conditions  under  which  this  strategy  is  reliable  are  given 
and  characterized.  A  by-product  of  this  domein  strategy  is  a  partial,  ability  to  de¬ 
tect  computation  errors.  It  is  the  objective  of  this  study  to  provide  an  analytical 
foundation  upon  which  to  base  practical  testing  implementations. 

There  are  limitations  inherent  to  any  testing  strategy,  and  these  also  constrain 
the  proposed  domain  strategy.  One  such  limitation  might  be  termed  colnc ldental 
correctness,  which  occurs  when  a  specific  test  point  follows  an  incorrect  path, 
and  yet  the  output  variables  coincidentally  are  the  same  as  if  that  test  point  were 
to  follow  the  correct  path.  This  test  point  would  then  be  of  no  assistance  in  the 
detection  of  the  domain  error  which  caused  the  control  flow  change.  No  test  gener¬ 
ation  strategy  can  circumvent  this  problem.  Another  inherent  testing  limitation  has 
been  previously  identified  as  a  missing  path  error ,  in  which  a  required  predicate 
does  not  appear  in  the  given  program  to  be  tested.  Especially  if  this  predicate  were 
an  equality,  no  testing  strategy  could  systematically  determine  that  such  a  predicate 
should  be  present. 

The  control  flow  statements  in  a  computer  program  partition  the  input  space  into 
a  set  of  mutually  exclusive  doma ins ,  each  of  which  corresponds  to  a  particular  pro¬ 
gram  path  and  consists  of  input  data  points  which  cause  that  path  to  be  executed. 

The  testing  strategy  generates  test  points  to  examine  the  boundaries  of  a  domain  to 
detect  whether  a  domain  error  has  occurred,  as  either  one  or  more  of  these  boundaries 
will  have  shifted  or  else  the  corresponding  predicate  relational  operator  has  changed. 
If  test  points  can  be  chosen  within  c  of  each  boundary,  the  strategy  is  shown  to 
be  reliable  in  detecting  domain  errors  of  magnitude  greater  than  c,  subject  to  the 
following  assumptions: 

(1)  coincidental  correctness  does  not  occur; 

(2)  missing  path  errors  do  not  occur; 

(3)  predicates  are  linear  in  the  Input  variables; 

(4)  the  input  space  is  continuous. 

Assumptions  (1)  and  (2)  have  been  shown  to  be  inherent  to  the  testing  process, 
and  cannot  be  entirely  eliminated.  However,  recognition  of  these  potential  problems 
can  lead  to  Improved  testing  techniques.  The  domain  testing  method  has  been  shown 
to  be  applicable  for  nonlinear  boundaries,  but  the  number  of  required  test  points 
may  become  Inordinate  and  there  are  complex  problems  associated  with  processing  non - 
linear  boundaries  In  higher  dimensions.  The  continuous  Input  space  assumption  Is 
not  really  a  limitation  of  the  proposed  testing  method,  but  allows  the  parameter  c 
to  be  chosen  arbitrarily  small.  An  error  analysis  for  discrete  spaces  is  available 
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and  Che  testing  strategy  has  been  proved  viable  as  long  as  the  size  of  the  domain  is 
not  comparable  to  the  discrete  resolution  of  the  space. 

Next  let  us  consider  two  further  assumptions: 

(5)  predicates  are  simple;  and 

(6)  adjacent  domains  compute  different  functions. 

If  assumptions  (5)  and  (6)  are  Imposed,  Che  testing  strategy  is  considerably 
simplified,  as  no  more  than  one  domain  need  be  examined  at  one  time  in  order  to 
select  test  polncs.  Moreover,  Che  number  of  test  points  required  Co  test  each 
domain  grows  linearly  with  both  the  dimensionality  of  the  input  space  and  the 
number  of  predicates  along  the  path  being  tested. 

The  only  completely  effective  testing  strategy  is  an  exhaustive  test  which  is 
totally  impractical.  The  domain  testing  strategy  offers  a  substantial  reduction  in 
the  high  cost  of  computer  program  testing,  and  yet  can  reliably  detect  a  major  class 
of  errors  which  have  been  characterized.  In  addition,  other  types  of  errors  can  be 
detected,  such  as  computation  errors  and  missing  path  errors,  but  this  detection 
cannot  be  guaranteed. 

The  domain  strategy  is  currently  being  implemented,  and  will  be  utilized  as  an 
experimental  facility  for  subsequent  research.  A  most  important  contribution  would 
be  to  indicate  both  programming  language  constructs  and  programming  techniques  which 
are  easier  to  test,  and  thus  produce  more  reliable  software. 
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CHAPTER  1 


INTRODUCTION 

Program  tearing  la  an  Inherently  practical  activity,  since  every 
computer  program  must  be  tested  before  any  confidence  can  be  gained  that 
the  program  performs  Its  Intended  function.  Some  of  the  best  designed 
software  has  required  that  nearly  as  much  effort  be  spent  planning  and 
Implementing  the  testing  process  as  was  Invested  In  the  actual  coding. 

What  the  practitioner  needs  are  better  guidelines  and  systematic  approaches 
In  the  design  of  the  testing  process  to  replace  the  ad  hoc  approach  which 
Is  now  so  prevalent  In  the  testing  of  computer  software. 

It  would  be  Ideal  If  there  existed  a  "theory  of  testing"  which  could 
be  used  to  rigorously  select  program  test  points.  The  problem  has  unfort¬ 
unately  proven  so  Intractable  that  no  comprehensive  testing  theory  exists. 

Research  by  Goodenough  and  Gerhart  [7]  and  Howden  [8,9]  has  resulted  In  an 
accepted  body  of  theory  concerning  testing,  and  has  provided  a  rigorous  basis 
for  further  research  In  this  area. 

The  objective  of  this  paper  Is  to  present  s  methodology  for  the  automatic 
selection  of  test  data.  Under  appropriate  assusiptlons,  this  methodology  will 
generate  test  data  which  will  detect  a  particular  class  of  errors  In  a 
program,  viz.,  "domain  errors"  as  defined  by  Howden  [9).  The  proposed  metho¬ 
dology  is  slso  described  In  greater  detail  In  Cohen  and  White  [3]  and  In  Cohen  [4]. 

The  goal  of  the  testing  process  Is  limited  to  the  successful  detection  of 
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a  program  arror  If  any  exists.  Any  attempt  to  identify  the  error,  Its  cause, 
or  an  appropriate  correction  la  properly  categorized  as  debugging .  and  Is 
beyond  the  scope  of  our  goal  In  the  testing  process.  Thus  testing  Is  essen¬ 
tially  error  detection,  while  debugging  Is  the  more  difficult  process  of 
error  correction.  Of  course.  In  practice  these  two  activities  usually 
overlap  and  are  frequently  combined  into  a  single  testing/debugging  phase  In 
the  software  development  cycle. 

An  Important  assumption  In  our  work  Is  that  the  user  (or  an  "erasle”) 

Is  available  who  can  decide  unequivocally  if  the  output  Is  correct  for  the 
specific  Input  processed.  The  oracle  decides  only  if  the  output  values 
are  correct,  and  not  whether  they  are  computed  correctly.  If  they  are 
incorrect,  the  oracle  does  not  provide  any  information  about  the  error 
and  does  not  give  the  correct  output  values. 

Tha  organization  of  the  report  Is  as  follows.  In  Chapter  2,  some 
preliminary  concepts  are  defined  and  discussed.  Some  assumptions  must 
be  made  concerning  the  language  In  which  the  given  computer  program  Is 
written,  and  the  ramifications  of  certain  language  constructs  are  explored. 
The  Important  concepts  of  program  path  and  path  psedlcates,  together 
with  domains,  are  defined  and  characterized.  The  case  of  linear 
predicates  Is  given  particular  emphasis,  since,  in  that  situation,  the 
domains  assume  the  simple  form  of  convex  polyhedra  In  the  Input  space. 

Logical  errors  In  a  computer  program  can  be  viewed  as  belonging  to 
one  of  two  classes  of  errors,  viz.,  "domain  errors”  and  "computation 
errors”.  Informally,  a  omaln  error  occurs  when  a  specific  Input  follows 
the  wrong  path  due  to  an  error  i  the  control  flow  of  the  program.  A  path 
contains  a  computation  error  when  a  specific  Input  follows  the  correct 
path,  but  an  error  In  some  assignment  statement  causes  the  wrong  function 
to  be  computed  for  one  or  more  of  the  output  variables. 
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The  third  chapter  rigorously  defines  these  error  classes,  and  explores 
the  ways  in  which  they  might  arise.  The  proposed  methodology,  called  the 
doeuln  strategy,  is  designed  specifically  to  detect  domain  errors.  In  this 
chapter,  we  will  discuss  two  fundamental  limitations  inherent  Co  any  finite  test 
strategy.  Once  such  limitation  might  be  termed  coincidental  correctness . 

This  occurs  when  the  computation  for  a  specific  test  point  is  Incorrect,  but 
the  output  value  happens  to  coincide  with  the  correct  value.  This  test  point 
would  then  be  of  no  assistance  in  the  detection  of  the  domain  error  which 
caused  the  change  in  control  flow.  Another  Inherent  testing  limitation  has 
been  identified  by  Howden  [9],  and  might  be  called  a  missing  path  error,  in 
which  a  required  predicate  does  not  appear  in  the  given  program  to  be  tested. 
This  could  result  in  a  situation  where  no  testing  strategy  can  systematically 
determine  that  such  a  predicate  should  be  present. 

The  domain  strategy  is  examined  in  Chapters  4  ana  5.  Inis  strategy  is 

developed  by  utilizing  the  structure  of  the  input  space  corresponding  to  the 
program.  More  specifically,  the  control  flow  partitions  the  input  space  into 
a  set  of  mutually  exclusive  domains.  Each  domain  corresponds  to  a  particular 
path  in  the  program  in  the  sense  that  the  set  of  input  data  points  in  that 
domain  will  cause  the  corresponding  path  to  be  executed.  The  strategy  proposed 
is  path-oriented;  in  testing  a  particular  path,  we  are  acutally  testing  the 
computations  performed  by  the  program  over  a  specific  Input  space  domain. 

Given  a  particular  path,  the  form  of  the  boundary  of  the  corresponding 
domain  Is  completely  determined  by  the  predicates  in  the  control  statements 
encountered  In  the  path.  Thus,  an  error  In  auch  a  predicate  will  be 
reflected  as  a  shift  In  the  boundary  of  the  corresponding  domain.  The 
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test lng  strategy  to  be  described  tests  a  path  for  domain  errors,  l.e.,  detects 
domain  boundary  shifts  by  observing  the  output  values  for  a  finite  number  of 
test  data  having  a  prescribed  geometrical  relationship  to  the  entire  domain 

and  Its  boundary.  These  output  values  are  computed  by  executing  the 

sequence  of  assignment  statements  constituting  the  path.  The  method  requires 
no  Information  other  than  the  successfully  compiled  program  for  selecting 
effective  test  data.  Thus  the  problem  has  been  converted  from  Its  usual  form  as 
an  Informal  studv  of  programs  and  programming  to  a  more  formal  Investigation 
of  the  geometry  of  Input  space  domains. 

The  strategy  Is  Initially  described  for  the  case  of  linear  predicates 
and  a  two-dimensional  Input  sp«.  e.  For  the  linear  case.  It  Is  shown  that, 
under  appropriate  assumptions,  the  number  of  test  points  to  reliably  test  a 
domain  grows  only  linearly  with  the  number  of  predicates  along  the  path  and 
with  the  dimensionality.  The  techniques  are  tnen  extended  to  N  dimensions, 
and  various  other  extensions  are  considered.  Including  nonlinear  predicates. 

A  domain  boundary  error  analysis  is  presented  In  Chapter  6,  which  Is  helpful 
In  choosing  the  best  locations  for  test  points.  The  application  uf  the  domain 
strategy  In  discrete  spaces  Is  analyzed  to  study  the  effect  of  roundoff  error 
In  selecting  test  points. 

In  the  concluding  Chapter  7  a  number  of  open  questions  generated  by  this 
Investigation  are  presented,  and  the  prospects  for  the  practical  application 
of  the  domain  testing  strategy  are  evaluated. 
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BACKGROUND  AND  PRELIMINARIES 
2 . 1  Programming  Language  Assumptions 

In  order  to  Investigate  domain  errors,  we  need  to  consider  the  language 
In  which  programs  will  be  written.  The  control  structures  should  be  simple 
and  concise,  and  should  resemble  those  available  In  most  procedure-oriented 
languages.  For  simplicity  we  assume  a  single  real-valued  data  type,  and  this 
Is  converted  to  Integer  values  for  use  as  DO-loop  Indices.  Because  this 
is  a  path-oriented  approach,  no  extra  control  flow  problems  are  introduced  by 
block  structure.  Thus  no  provision  is  made  for  block  structure,  as  It  would 
only  add  extra  bookkeeping  to  keep  track  of  local  variables  and  block 
Invocation  or  exit. 

A  number  of  programming  language  features  are  assumed  not  to  occur  In  the 
programs  we  are  to  analyze  for  domain  errors.  The  first  feature  Is  that  of 
arrays;  despite  the  fact  that  arrays  consnonly  occur  in  programs,  a  predicate 
which  refers  to  an  element  of  an  Input  array  can  cause  major  complications 
(Ramamoorthy  [11]).  A  second  class  of  language  features  which  will  be  excluded 
In  our  analysis  Is  that  of  subroutines  and  functions.  The  problems  of  side 
effects  and  of  parameter  passing  pose  difficulties  for  domain  testing.  The 
third  class  of  features  which  are  not  currently  analyzed  by  domain  testing 
Include  nonnuasrlcal  data  types  such  as  character  data  and  pointers.  These 
are  admittedly  very  Important  features,  and  further  research  Is  needed  to 
Investigate  whether  these  features  pose  any  fundamental  limitations  to  the 
domain  testing  strategy. 

Since  Input/output  processing  Is  so  closely  linked  to  a  machine  or  compiler 
environment,  we  will  assume  that  all  I/O  errors  have  previously  been  eliminated. 
Thus  only  the  most  elementary  I/O  capabilities  are  provided;  Input  Is  provided 
by  a  simple  READ  statement,  and  output  Is  accomplished  with  a  simple  WRITE 


statement . 
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The  types  of  control  flow  constructs  Investigated  in  this  research  include 
sequence,  alternation,  and  Iteration  control.  Since  the  analysis  is  path- 
oriented,  GO-TO  statements  could  be  included  without  adversely  affecting  any 
results,  except  that  program  paths  could  become  quite  complex. 

All  computation  is  ac  ompllshed  by  means  of  arithmetic  assignment  state¬ 
ments  which  also  provide  the  basic  sequential  flow  of  control.  In  each 
statement  a  single  variable  is  assigned  a  value.  The  right  hand  side  of  an 
assignment  statement  is  an  arithmetic  expression  using  variables,  constants, 
and  a  set  of  basic  arithmetic  operators  (+,  -,  *,  /). 

The  general  predicate  form  used  for  control  flow  is  a  Boolean  combination 
of  arithmetic  relational  expressions.  The  logical  operators  OR  and  AND  are 
used  to  form  these  Boolean  combinations.  Each  arithmetic  relational  expression 
contains  a  relational  operator  from  the  set  (<,  >,  *,  >,  +) .  These  operators 

form  a  complete  set,  and  thus  the  logical  operator  NOT  is  unnecessary.  If  a 
predicate  consists  of  two  or  sore  relational  expressions  with  Boolean  operators, 
then  it  is  a  compound  predicate .  A  simple  predicate  consists  of  Just  a  single 
relational  expression. 

The  alternation  type  of  control  flow  is  achieved  by  using  the  IF-THEN- 
ELSE-ENDIF  construct.  The  conditional  associated  with  the  IF  statement  is  a 
general  predicate.  Any  well-formed  program  segment,  including  the  null  program 
segment,  can  be  used  in  the  THEN  and  ELSE  portions  of  the  IF  construct.  The 
ENDIF  statement  is  Just  a  delimiter  for  the  IF  construct,  which  clarifies 
the  nesting  structure  and  eliminates  any  potentially  ambiguous  ELSE  clause. 

A  general  iteration  construct  is  Included  which  consists  of  a  DO 
statement,  loop  body,  and  ENDDO  delimiter.  The  DO  statement  can  be  In  one  of 
three  forms: 

1)  DO  I  -  INIT,  FINAL,  1NCR; 

2)  DO  WHILE  (general  predicate); 

3)  DO  I  ■  INIT,  FINAL,  INCR  WHILE  (general  predicate). 
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The  loop  body  can  be  any  well-formed  program  segment,  and  the  ENDDO  Is  Just  a 

delimiter  to  clarify  the  scope  of  the  Iteration. 

The  variables  used  in  a  program  are  divided  into  three  classes.  If  a  variable 

appears  in  a  READ  or  WRITE  statement,  it  is  classified  as  an  Input  or  output 

variable  respectively;  all  other  variables  are  called  program  variables . 

In  order  to  produce  a  clear  delineation  between  the  three  types  of  variables, 

we  assume  that  a  given  variable  belongs  to  only  one  of  the  above  three  classes. 


1 . 2  Program  Paths  and  Path  Predicates 

A  program  can  be  represented  as  a  directed  graph  G  ■  (V,A),  where  V  is 
a  sec  of  nodee  and  A  is  the  set  of  arcs  or  directed  edges  between  nodes.  In 

the  language  discussed  In  Section  2,1,  we  have  defined  a  set  of  haelc  prograi* 
elements  which  consists  of  a  READ,  WRITE,  assignment,  IF,  and  DO  statement, 

together  with  the  ENDIF  and  ENDDO  delimiters.  The  directed  graph  representation 
of  a  program  will  contain  a  node  for  each  occurrence  of  a  basic  program  element, 
and  an  arc  for  each  possible  flow  of  control  between  these  elements.  While  THEN 
and  ELSE  statements  do  not  explicitly  appear  in  the  digraph,  the  action* 
associated  with  them  will  be  represented  as  nodes  In  the  digraph. 

A  walk  In  a  digraph  Is  defined  as  an  alternating  sequenc  2  nodes  and 

arcs  (v^  A^j  v2  A23  . *k-l  k  Vk^  euch  that  each  arc  A^^^  La  directed  irua 

node  v^  to  node  A  control  path  is  than  defined  to  be  a  walk  In  the  directed 

graph  beginning  with  the  node  for  the  Initial  statement  and  terminating  with  the 
node  for  Che  final  statement.  It  should  be  noted  that  two  walks  which  differ 
only  In  the  number  of  times  a  particular  loop  in  the  program  is  executed 
will  be  defined  as  two  distinct  control  paths.  Thus  the  number  of  control  paths 
In  a  program  can  be  Infinite. 

Every  branch  point  of  the  program  Is  associated  with  a  general  predicate. 

This  predicate  evaluates  to  true  or  false,  and  Its  value  determines  which  outcome 
of  the  branch  will  be  followed.  A  predicate  Is  generated  each  time  control 
reaches  an  IF  or  DO  statement  in  the  given  languege.  The  peth  condition  Is  the 
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compound  condition  which  must  be  satisfied  by  the  input  data  point  in  order  for  the 
control  path  to  be  executed.  It  is  the  conjunction  of  the  individual  predicate 
conditions  which  are  generated  at  each  branch  point  along  the  control  path. 

Not  all  the  control  paths  that  exist  syntactically  within  the  program  are 

executable.  If  input  data  exist  which  satisfy  the  path  condition,  the  control 
path  is  also  an  execution  path  and  can  be  used  in  testing  the  program.  If  the 

path  condition  is  not  satisfied  by  any  input  value,  the  path  is  said  to  be 

infeasible,  and  is  of  no  use  in  testing  the  program. 

A  simple  predicate  is  said  to  be  linear  in  variables  V, ,  V, . V 

■  -  1  Z  n 

if  it  is  of  ,  the  form 


A,  V,  A,V.  ♦  ....  +  A  V  ROP  K, 

112  2  n  n  • 

where  K  and  the  A^  are  constants,  and  ROP  represents  one  of  the  relational 
operators  .  A  compound  predicate  is  linear  when  each  of  its 


component  simple  predicates  is  linear. 


In  general,  predicates  can  be  expressed  in  terms  of  both  program  variables 
and  input  variables.  However,  in  generating  input  data  to  satisfy  the  path 
condition  we  must  work  with  constraints  in  terms  of  only  input  variables. 


If  we  replace  each  program  variable  appearing  in  the  predicate  by  its  symbolic 
value  in  terms  of  input  variables,  we  get  an  equivalent  constraint  which  we 
call  the  predicate  Interpretation.  A  particular  interpretation  is  equivalent 
to  the  original  predicate  in  that  input  variable  values  satisfying  the  inter¬ 
pretation  will  lead  to  the  computation  of  program  variables  which  also  satisfy 


the  original  predicate.  A  single  predicate  can  have  many  different  interpre¬ 
tations  depending  upon  which  path  is  selected,  for  each  path  will  in  general 
consist  of  a  different  sequence  of  assignment  statements.  The  following 


program  segment  provides  example  predicates  and  interpretations. 
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READ  A, B; 

IF  A  >  B 

THEN  C  -  B  +  1; 
ELSE  C  -  B  -  1; 
ENDIF; 

D  -  2*A  +  B; 

IF  C  <  0 
THEN  E  -  0; 

ELSE 

DO  I  -  l.B; 

E  -  E  +  2*1; 
END DO; 

ENDIF; 

IF  D  -  2 

THEN  F  -  E  +  A; 
ELSE  F  -  E  -  A; 
ENDIF; 

WRITE  F; 


In  Che  flrsC  predicate,  A  >  B,  both  A  and  B  are  Input  variables,  so  there 
Is  only  one  Interpretation.  The  second  predicate,  C  <_  0,  will  have  two 
Interpretations  depending  on  which  branch  was  taken  In  the  first  IF  construct. 

For  paths  on  which  the  THEN  C  •  J  +  1  clause  Is  executed  the  interpretation  Is 

B  +  1  <_  0  or  equivalently  B  <_  -1.  When  the  ELSE  C  -  B  -  1  branch 

Is  taken,  the  Interpretation  Is  B  -  1  <  0,  or  equivalently  B  1 .  Within 

the  second  IF-THEN-ELSE  clause,  a  nested  DO-loop  appears.  The  DO-loop  Is 

executed : 

no  time 3  If  B  <  1 

once  If  1  ^  B  <  2 
twice  If  2  ^  B  <  3 
etc. 

Thus  the  selection  of  a  path  will  require  a  specification  of  the  number  of  times 
that  the  DO-loop  Is  executed,  and  a  corresponding  predicate  Is  applied  which 
selects  those  Input  points  which  will  follow  that  particular  path.  Even  though 
the  third  predicate,  D  -  2,  appears  on  four  different  paths.  It  only  has  one 
interpretation,  2*A  +  B  ■  2,  since  D  is  assigned  the  value  2*A  +  B  In  the 
same  statement  In  each  of  the  four  paths. 


The  domain  testing  strategy  becomes  particularly  attractive  from  a 


practical  point  of  view  if  the  predicates  are  assumed  to  be  linear  in  input 
variables.  It  might  seem  to  be  an  undue  limitation  to  require  that  predicate 
Interpretations  be  linear  for  the  proposed  strategy.  In  fact,  however,  as  the 
following  discussion  shows,  this  represents  no  real  limitation  for  many 
Important  applications. 

A  number  of  authors  have  provided  data  to  show  that  simple  progransnlng 
language  constructs  are  used  more  often  than  complex  constructs.  Knuth  (10] 
studied  a  random  sample  of  FORTRAN  programs  and  found  that  86*  of  all  assign¬ 
ment  statements  were  of  the  forms 


V  ■  V 

1  2’ 

VI  ‘  V2  +  V 

or  Vx  -  V2  -  Vj. 

Also  70?  of  all  DO  loops  in  the  programs  contained  less  than  four  statements. 
Elshoff  [5,6]  studied  120  production  PL/I  programs  and  showed  similar  results, 
including  the  fact  that  97J  of  all  arithmetic  operators  are  ♦  or  -,  and  98? 
of  all  expressions  contain  fewer  than  two  operators. 

An  experiment  of  particular  relevance  to  the  present  context  is  reported 
in  Cohen  (4]  using  typical  data  processing  programs,  since  program  functions 
and  programming  practice  tend  to  be  reasonably  uniform  in  this  area.  A  random 
sample  of  50  COBOL  programs  was  taken  directly  from  production  data  processing 
applications  for  this  study.  In  this  static  analysis  each  predicate  is 
classified  according  to  whether  it  is  linear  or  nonlinear,  and  the  number  of 
input  variables  used  In  the  predicate  has  also  been  recorded.  In  addition,  the 
number  of  input-independent  predicates  were  tabulated,  since  these  predicates 
do  not  produce  any  input  constraints.  The  number  of  equality  predicates  is 
also  reported  since  these  predicates  are  very  beneficial  in  reducing  the  number 
of  test  points  required  for  a  domain.  These  data  are  summarized  in  Table  I. 
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TOTAL 

AVG. 

RANGE 

Total  Lines 

12,628 

253 

31-1,287 

Procedure  Division  Lines 

8,139 

163 

13-822 

Total  Predicates 

1,225 

25 

0-115 

Linear  Predicates 

1,070 

21 

0-104 

Nonlinear  Predicates 

1 

0.02 

0-1 

Input-Independent  Predicates 

154 

3 

0-28 

Predicates  with  1  Variable 

945 

19 

0-97 

Predicates  with  2  Variables 

125 

2.5 

0-20 

Equality  Predicates 

779 

15.5 

0-76 

TABLE  I  Predicate  Statistics  for  50  COBOL  Programs 
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The  most  Important  result  Is  that  only  one  predicate  out  of  the  1225 
tabulated  In  the  study  can  possibly  be  a  nonlinear  predicate.  The  predicates 
are  also  very  simple  since  most  of  them  refer  to  only  one  Input  variable,  and 
no  predicate  In  this  sample  uses  more  than  two  Input  variables. 

In  conclusion,  while  this  study  by  no  means  represents  an  exhaustive 
survey,  we  believe  the  sample  Is  large  enough  to  Indicate  that  nonlinear 
predicate  Interpretations  are  rarely  encountered  In  data  processing  applications. 
It  Is  clear  that  any  testing  strategy  restricted  to  linear  predicates  Is  still 
viable  In  many  areas  of  programing  practice. 

2.4  Input  Scats  Structure 

A  program  which  has  N  Input  variables  and  produces  M  output  variables 
computes  a  function  which  maps  points  In  the  N-d lmenslonal  Input  space  to 
points  In  the  M-dlaenslonal  output  space.  The  Input  space  Is  partitioned  Into 
a  set  of  domains.  Each  domain  corresponds  to  a  particular  executable  path  In 
the  program  and  consists  of  the  Input  data  points  which  cause  the  path  to  be 
executed.  More  formally,  an  Input  space  domain  la  defined  as  a  set  of  Input 
data  points  satisfying  a  path  condition,  consisting  of  a  conjunction  of  predi¬ 
cates  along  the  path.  In  this  discussion,  these  predicates  are  assumed  to  be 
simple;  compound  predicates  will  be  discussed  later  in  Section  5.3. 

We  assume  that  the  Input  space  Is  bounded  In  each  direction  by  the 
minimum  and  maximum  values  for  the  corresponding  variable.  These  mln-max 
constraints  do  not  appear  In  the  program  but  are  automatically  appended  to 
each  path  condition.  Since  a  single  data  type  Is  used  for  all  variables  In 
our  language,  each  variable  will  have  the  same  mln-max  constraints. 

The  boundary  of  each  domain  la  determined  by  the  predicates  In  the  path 
condition  and  consists  of  border  segments .  where  each  segment  Is  the  section  of 
the  boundary  determined  by  a  single  simple  predicate  In  the  path  condition. 

Each  border  segment  can  be  open  or  closed  depending  on  the  relational  operator 
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in  the  predicate.  A  closed  bordar  segment  la  actually  pare  of  cha  domain  and 
la  formad  by  predlcataa  with  <_,  > ,  or  ■  oparators.  An  opan  bordar  segment  forma 
part  of  cha  domain  boundary  but  doaa  not  constitute  part  of  tha  domain,  and 
la  formad  by  <,  >,  and  i  pradlcatas.  We  shall  find  It  convenient  to  use  the 
term  border  operator  to  refer  to  the  relational  operator  for  tha  corresponding 
predicate . 

Since  bordar  segments  in  tha  Input  space  are  determined  by  tha  particular 
predicate  Interpretations  on  tha  path,  tha  form  of  tha  segment  may  be  different 
from  that  of  the  original  predicate.  For  example,  with  input  variablas  A  and  B, 
the  linear  predicate  A <  C  ♦  2  can  lead  to  a  nonlinear  border  segment,  A<  B*B  ♦  2, 
when  C  -  B*B .  Similarly,  a  nonlinear  predicate,  C  >  A*A  +  B,  will  produce 
a  linear  border  segment,  A  ^  B,  when  C  -  A*A  +  A.  Since  a  predicate  can  appear 
on  many  paths  and  each  path  can  execute  a  different  sequence  of  assignment 
statements  for  the  variables  used  in  the  predicate,  a  single  predicate  can  have 
many  different  interpretations  and  can  form  many  discontinuous  border  segments 
for  various  domains. 

The  total  number  of  predicates  on  the  path  is  only  an  upper  bound  on 
the  number  of  border  segments  in  the  domain  boundary  since  certain  predicates 
in  the  path  condition  may  not  actually  produce  border  segments.  An  input- 
independent  predicate  interpretation  is  one  which  reduces  to  a  relation  between 
constants,  and  since  it  is  either  true  or  false  regardless  of  the  input  values, 
it  does  not  further  constrain  the  domain.  A  redundant  predicate  interpretation 
is  one  which  is  superceded  by  the  other  predicate  interpretations,  l.e.,  the 
domain  can  be  defined  by  a  strict  subset  of  the  predicate  interpretations  for 
that  path. 

The  general  form  of  a  simple  linear  predicate  interpretation  la 

A.  X,  ♦  A,  X,  ♦  ....  ♦  A  X  HOP  X 
112  2  n  n 
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where  ROP  Is  the  relational  operator,  are  input  variables,  and 

A^,  X  are  constants.  However,  the  border  segment  which  any  of 

these  predicates  defines  Is  a  section  of  the  surface  defined  by  the  equality 

A,  X,  +  A,  X,  ♦ _ +  A  X  -  K, 

112  2  n  n 

since  this  Is  the  Halting  condition  for  the  points  satisfying  the  predicate. 

In  an  N-dlmensional  space  this  linear  equality  defines  a  hyperplane  which  Is 
the  N-dlaenslonal  generalization  of  a  plane. 

Consider  a  path  condition  composed  of  a  conjunction  of  simple  predicates. 

These  predicates  can  be  of  three  basic  types:  equalities  (■) ,  Inequalities  (< , 

>,  *_) ,  and  nonequalities  («*) .  The  use  of  each  of  the  three  types  results  in  a 

markedly  different  effect  on  the  domain  boundary.  Each  eouality  constrains  the  domain 
to  lie  In  a  particular  hyperplane,  thus  reducing  the  dimensionality  of  the 
domain  by  one.  The  set  of  Inequality  constraints  then  defines  a  region  within 
the  lower  dimensional  space  defined  by  the  equality  predicates. 

The  nonequality  linear  constraints  define  hyperplanes  which  are  not  part 
of  the  domain,  giving  rise  to  open  border  segments  as  mentioned  earlier.  Obaerve 
that  the  constraint  A  4  B  is  equivalent  to  the  compound  predicate  (A  <  B)  OR 
(A  > B) .  In  this  form  It  Is  clear  that  the  addition  of  a  nonequality  predicate 
to  a  set  of  Inequalities  can  split  the  domain  defined  by  those  inequalities  Into 
two  regions. 

The  following  example  should  clarify  the  concepts  discussed  above, 

READ  I.J; 

C  -  I  +  2*J  -  1; 

(PI)  IP  C  >  6 

THEN  D  -  C  -  I; 

ELSE  D  -  C  I  -  J  +  2; 

ENDIP; 

(P2)  IF  D  •  C  ♦  2 
THEN  E  -  I; 

ELSE  E  -  3; 

ENDIP; 

(P  3)  IP  E  <  D  -  2*J 
THEN  F  -  I; 

ELSE  F  -  J; 

HID  IF; 


WRITE  F; 
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Figure  1  shows  ths  corresponding  input  space  partitioning  structure  for 
this  program.  The  input  space  is  in  terms  of  Inputs  I  and  J,  and  is  arbitrarily 
constrained  by  the  following  min-max  conditions; 

-J  <  i  <  4,  -2  I  J  1  6. 

Each  border  in  Figure  1  is  labelled  with  the  corresponding  predicate,  and  each 
domain  is  labelled  with  the  corresponding  path.  The  path  notation  is  based 
upon  which  branch  (T  or  E)  Is  taken  in  each  of  the  three  IF  constructs,  e.g.,  TEE. 

The  first  predicate  PI,  C  >  6,  will  be  interpreted  as  I  +  2*J  >  7  since 
C  •  I  +  2*J  -  1.  This  single  interpretation  PI  is  seen  in  Figure  1  as  a  single 
continuous  border  segment  across  the  entire  input  space.  The  second  predicate 
P2  demonstrates  the  effects  of  both  equality  and  nonequality  predicates.  Domains 
for  paths  through  the  THEN  branch  are  constrained  by  the  equality,  and  this 
reduction  in  dimensionality  is  seen  in  the  fact  that  these  domains  consist  of 
the  points  on  the  solid  line  segments  ETT  and  TTT.  Paths  through  the  ELSE 
branch  are  constrained  by  a  nonequality  predicate,  and  the  corresponding  domains 
consist  of  the  two  regions  on  either  side  of  the  solid  line  segments  (e.g.,  EEE) . 
This  predicate  has  two  interpretations  depending  upon  the  value  assigned  to  D 
and  produces  two  discontinuous  border  segments  ETT  and  TTT. 

The  third  predicate  P3  might  have  four  different  interpretations,  but 
only  one  border  segment  appears  In  the  diagram.  The  other  three  interpretations 
do  not  produce  borders  since  they  are  either  redundant,  Input-Independent,  or 
correspond  to  infeasible  paths.  With  three  IF  constructs  we  have  eight  control 
paths,  but  the  diagram  contains  only  five  domains  since  three  of  the  paths  are 
infeasible.  Also  many  of  these  domains  have  fewer  than  three  border  segments 
because  of  redundant  and  input- Independent  interpretations.  From  this  example  we 
can  conclude  that  the  input  space  partitioning  structure  of  a  program  with  many 
predicates  and  a  larger  dimensional  input  space  can  be  extremely  complicated. 
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The  foregoing  definitions  and  the  example  allow  us  to  characterize  more 

precisely  domains  which  correspond  to  simple  linear  predicate  Interpretations. 

For  a  formal  statement  of  the  characterisation,  we  need  the  following  definitions. 

A  set  is  convex.  If  for  any  two  points  In  the  set,  the  line  segment  joining 

these  points  Is  also  in  the  set.  A  convex  polyhedron  Is  the  set  produced  by  the 

intersection  of  the  set  of  points  satisfying  a  finite  number  of  linear  equalities  and 
inequalities. 

Proposition  1 

For  an  execution  peth  with  a  set  of  simple  linear  equality  or  Inequality 
predicate  Interpretations,  the  input  space  domain  Is  a  single  convex  polyhedron. 

If  one  or  more  simple  linear  nonequality  predicate  interpretations  are  added  to 
this  set,  then  the  input  space  domain  consists  of  the  union  of  a  set  of  disjoint 
convex  polyhedra. 


CHAPTER  3 
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ERROR  CLASSIFICATION  AND  THEORETICAL  LIMITATIONS 
3.1  Definitions  of  Types  of  Error 

The  beslc  Ideas  behind  the  classification  of  errors  that  we  use  are  due  to 
Hovden  [9],  but  our  approach  to  defining  them  Is  somewhat  more  operational 
than  that  given  In  his  paper. 

From  the  previous  sections.  It  Is  clear  that  a  program  can  be  viewed  as 

1)  establishing  an  exhaustive  partition  of  the  Input  space 

Into  mutually  exclusive  domains  each  of  which  corresponds 
to  an  executable  path,  and 

2)  specifying,  for  each  domain,  a  set  of  assignment  statements 
which  constitute  the  domain  computation. 

Thus  we  have  a  canonical  representation  of  a  program,  which  is  a  (possibly 
Infinite)  set  of  pairs  {  (Dx  ;f  ^  ,  (D2 ;  f  2> ,  ...  (D^f^,..),  where  Is  the  i-th 
domain,  and  f^  is  the  corresponding  domain  computation  function. 

Given  an  Incorrect  program  P,  let  us  consider  the  changes  in  Its 
canonical  representation  as  a  result  of  modifications  performed  on  P.  It  is 
assumed  that  these  modifications  are  made  using  only  permissible  language 
constructs  and  results  in  a  legal  program. 

Definition:  A  domain  boundary  modification  occurs  If  the  modification 
results  In  a  change  In  the  D^  component  of  some  pair  in  the  canonical 

representation. 

Definition:  A  domain  computation  mod  if lcatlon  occurs  If  the  modification 
results  In  a  change  in  the  component  of  some  P«i*  in  the  canonical 


representation. 
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Definition:  A  aliasing  path  modlf lcatlon  occurs  If  the  modification  results  in 
the  creation  of  a  new  (D^f^)  pair  such  that  is  a  subset  of  occurring  In  some 
pair  canonical  representation  of  P,  and  f^  differs  from  f^. 

Notice  that  a  particular  modification  (say  a  change  of  some  assignment 
statement)  can  be  a  modification  of  more  than  one  type.  In  particular,  a 
missing  path  modification  is  also  a  domain  boundary  modification. 

The  errors  that  occur  In  a  program  can  be  classified  on  the  basis  of  the 
modifications  needed  to  obtain  a  correct  program  and  consequent  changes  in  the 
canonical  representation.  In  general,  there  will  be  many  correct  programs,  and 
multiple  ways  to  get  a  particular  correct  program.  Hence,  the  error  classifi¬ 
cation  Is  not  unique,  but  relative  to  the  particular  correct  program  that 
would  result  from  the  series  of  modif ications. 

Definition:  An  Incorrect  program  P  can  be  viewed  as  having  a  doma In  error 

* 

(computat Iona  1  error )  (missing  path  error )  If  a  correct  program  P  can  be 
created  by  a  sequence  of  modifications  at  least  one  of  which  is  a  domain 
boundary  modification  (domain  computation  modif lcat ion) (missing  path 
modlf lcatlon) . 

Several  remarks  are  in  order.  The  operational  consequence  of  the  phrase 
"can  be  viewed  as"  in  the  above  definition  Is  that  the  error  classification 

is  relative  not  only  to  a  particular  correct  program,  but  also  to  a  particular 
sequence  of  modifications.  For  Instance,  consider  an  error  In  a  predicate 
interpretation  such  that  an  Incorrect  relational  operator  Is  employed,  c.g.,  use 
of  >  instead  of  <.  This  could  be  viewed  as  a  domain  error,  leading  to  a 
modification  of  the  predicate,  or  as  a  computation  error,  leading  to  a  modification 
of  the  functions  computed  on  the  two  branches.  The  fact  that  it  might  be 
more  profitable  to  change  the  relational  operator  rather  than  the  function 
computations  is  a  consequence  of  the  language  constructs,  and  is  not  directly 
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captured  In  the  definitions  of  the  types  of  error.  In  this  paper  we  would 
regard  an  error  due  to  an  Incorrect  relational  operator  as  a  domain  error; 

It  Is  a  simpler  modification  to  change  the  relational  operator  In  the  predicate 
than  to  Interchange  the  set  of  assignment  statements. 

More  specific  characterizations  of  these  errors  can  be  made  in  the  context 
of  the  specific  programing  language  which  we  have  introduced.  In  particular, 
the  following  Informal  description  directly  relates  the  domain  and  missing 
path  errors  to  the  predicate  constructs  allowed  in  the  language. 

A  path  contains  a  domain  error  If  an  error  In  some  predicate  Interpre¬ 
tation  causes  a  border  segment  to  be  "shifted"  from  its  correct  position  or 
to  have  an  incorrect  border  operator.  A  domain  error  can  be  caused  by  an 
Incorrectly  specified  predicate  or  by  an  Incorrect  assignment  statement  which 
affects  a  variable  used  In  the  predicate.  An  Incorrect  predicate  or 
assignment  statement  may  affect  many  predicate  Interpretations  and  conse¬ 
quently  cause  more  than  one  border  to  be  In  error. 

A  path  contains  a  missing  path  error  when  a  predicate  is  missing  which 
would  subdivide  the  domain  and  create  a  new  execution  path  for  one  of  the 
subdomains.  This  type  of  error  occurs  when  some  special  condition  requiring 
different  processing  Is  omitted. 


3.2  Fundamental  Limitations 

Finite  testing  strategies  are  fundamentally  limited  by  their  Inability 
to  detect  phenomena  occurlng  In  regions  which  have  zero  volume  or  measure 
relative  to  the  Input  apace  or  domain.  The  first  of  these  limitations  we  shall 
define  as  coincidental  correctness.  In  testing  each  domain  for  the 


correctness  of  Its  boundaries.  If  the  output  for  a  test,  case  Is  correct,  It 
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could  be  either  that  the  test  point  was  In  the  correct  domain,  or  that  it  was 
in  a  wrong  domain  but  the  computation  in  that  domain  coincidentally  yielded 
a  correct  value  for  the  test  point.  Similarly,  a  domain  computation  could 
correspond  to  an  incorrect  function,  but  its  output  may  coincide  with  the 
correct  value  for  a  particular  test  point.  To  be  absolutely  certain  that  the  values 
are  not  coincidentally  correct,  it  would  be  necessary  to  exhaustively  test  all 
the  points  of  the  domain. 

The  essence  of  the  coincidental  correctness  problem  is  the  same  as 
that  of  the  problem  of  deciding  if  two  arbitrary  computations  are 
equivalent;  the  latter  problem  is  known  to  be  generally  undecidable.  However, 
in  practice,  the  severity  of  the  problem  is  related  to  the  probability  that 
for  an  arbitrary  point  this  coinc idence  would  occur.  If  the  set  of  points 
for  which  the  two  functions  have  the  same  value  is  of  measure  zero,  then  this 
probability  is  zero,  even  though  coincidental  correctness  is  still  possible. 

So,  even  with  coincidental  correctness  as  a  possibility,  a  testing  strategy 
can  be  almost  reliable  in  the  sense  of  Howden  [9],  if  it  would  be  reliable 
in  the  absence  of  coincidental  correctness,  and  the  set  of  points  which  are 
coincidentally  correct  has  zero  volume  relative  to  the  domain  being  tested. 

Another  basic  limitation  relates  to  missing  path  errors.  When  the 
subdomain  associated  with  a  missing  path  is  a  region  of  lower  dimensionality 
than  the  original  domain,  a  missing  path  error  of  reduced  dimensionality 
occurs.  This  typically  happens  when  the  missing  predicate  is  an  equality.  If 
all  that  is  available  is  Just  the  (incorrect)  program  to  be  tested,  then  the 
probability  that  a  finite  set  of  test  points  would  detect  the  missing  predicate 
is  zero,  since  the  volume  of  the  subdomain  is  zero  relative  to  that  of  the 


original  domain. 
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The  proposed  approach  ts  capable  of  detecting  many  kinds  of  missing  path 
errors,  but  for  some  of  them  the  number  of  required  test  points  is  Inordinate. 
Hence,  in  the  next  section,  where  we  describe  the  testing  strategy,  we  will 
simply  assume  that  no  missing  path  errors  are  associated  with  the  path  being 


tested . 


CHAPTER  4 
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THE  DOMAIN  TESTING  STRATECY 

The  domain  testing  strategy  is  designed  to  detect  domain  errors  and  will 
be  effective  in  detecting  errors  in  any  type  of  domain  border  under  certain 
conditions.  Test  points  are  generated  for  each  border  segment  which,  if 
processed  correctly,  determine  that  both  the  relational  operator  and  the 
position  of  the  border  are  correct.  An  error  in  the  border  operator 
occurs  when  an  Incorrect  relational  operator  is  used  in  the  corresponding 
predicate,  and  an  error  in  the  position  of  the  border  occurs  when  one  or  more 
incorrect  coefficients  are  computed  for  the  particular  predicate  Interpretation. 
The  strategy  is  based  on  a  geometrical  analysis  of  the  domain  boundary  and 
takes  advantage  of  the  fact  that  points  on  or  near  tne  border  are  most 
sensitive  to  domain  errors.  A  number  of  authors  have  made  this  observation, 
e.g.,  Bover  et  al.  fl]  and  Clarke  [2]. 

As  stated  in  Proposition  1,  a  domain  defined  by  simple  linear  predicates 
is  a  convex  polyhedron,  and  each  point  can  be  classified  according  to  its 
position  within  the  domain.  An  interior  point  is  defined  as  one  which  is 
surrounded  by  an  e -neighborhood  containing  only  points  in  the  domain. 

Similarly,  a  boundary  point  is  one  for  which  every  t -neighborhood  contains 
both  points  in  the  domain  and  points  lying  outside  of  the  domain.  Finally, 
an  extreme  point  is  a  boundary  point  which  does  not  lie  between  any  two 
distinct  points  in  the  domain. 

In  the  previous  section,  a  comparison  was  made  between  the  given  program  and 
corresponding  correct  program;  indeed  domain  errors  were  defined  in  terms 
of  this  correspondence.  It  should  be  emphasized  that  the  domain  strategy 
does  not  require  that  the  correct  program  be  given  for  the  selection  of  test 
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point s ,  since  only  information  obtained  from  the  given  program  la  needed. 
However,  it  will  be  convenient  to  be  able  to  refer  to  a  "correct  border' , 
although  it  will  not  be  necessary  to  have  any  knowledge  about  this  border. 
Define  the  given  border  as  that  corresponding  to  the  predicate  Interpretation 
for  the  given  program  being  tested,  and  the  correct  border  as  that  border 
which  ifould  be  calculated  in  some  correct  program. 

The  domain  testing  strategy  is  first  developed,  explained,  and  validated 
in  detail  under  a  set  of  simplifying  assumptions: 

1)  Coincidental  correctness  does  not  occur  for  any  test  case.  If 

correct  output  results  are  produced,  we  can  assume  that  the  test 
point  is  in  the  correct  domain  rather  than  bej.ng  coincidentally 
correct  in  another  domain. 

2)  A  missing  path  error  is  not  associated  with  the  path  being  tested. 

Missing  path  errors  of  reduced  dimensionality  pose  a  theoretical 
limitation  to  the  reliability  of  any  program  testing  methodology. 

3)  Each  border  is  produced  by  a  simple  predicate. 

4)  The  path  corresponding  to  each  adjacent  domain  computes  a  different 

function  than  the  path  being  tested. 

5)  The  given  border  is  linear,  and  if  it  is  incorrect,  the  correct 

border  is  also  linear. 

6)  The  input  space  is  continuous  rather  than  discrete. 

7)  Each  border  is  produced  by  an  inequality  predicate. 

8)  The  input  space  is  two-dimensional ,  corresponding  to  a  program  which 

reads  at  most  two  input  variables. 


The  first  two  assumptions  were  thoroughly  explored  In  the  previous  section. 
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Assumptions  3)  through  8)  arc  for  convenience  in  the  initial  exposition,  and 
we  shall  investigate  later  the  conditions  under  which  each  can  be  relaxed.  Also, 
references  (3)  and  [4|  discuss  both  the  domain  atrategy  and  these  assumptions 
In  greater  detail. 

The  Tvo-Dlmens tonal  Linear  Case 

Clven  assumptions  l)  -  8),  a  set  of  test  points  is  first  defined  for 
detecting  border  shifts,  and  then  we  shall  show  that  this  set  of  points  also 
detects  all  possible  relational  operator  errors.  Since  the  present  analysis 
is  limited  to  linear  borders  in  a  two-dimensional  input  space,  each  border  is 
a  line  segment.  Therefore,  the  correct  border  can  be  determined  if  we  know 
two  points  on  that  border. 

The  test  cases  selected  will  be  of  two  types,  defined  by  their  position 
with  respect  to  the  given  border.  An  ON  test  point  lies  on  the  given  border, 
while  an  OFT  test  point  is  a  small  distance  c  fromf  and  lies  on  the  open 

side  of,  the  given  border.  Therefore,  we  observe  that  when  testing  a  closed 
border,  the  ON  test  points  are  in  the  domain  being  tested,  and  each  OFF  test 
point  is  in  some  adjacent  domain.  Conversely,  when  testing  an  open  border, 
each  ON  test  point  is  in  some  adjacent  domain,  while  the  OFF  test  points  are 
in  the  domain  being  tested. 

Figure  2  shows  the  selection  of  three  test  points  A,  B,  and  C  for  a 
closed  inequality  border  segment.  In  this  and  subsequent  figures  the  small 
arrows  are  used  to  indicate  the  domain  which  contains  the  border  segment.  The 
three  points  must  be  selected  in  an  ON-OFF-ON  sequence.  Specif ically ,  if 
test  point  C  is  projected  down  on  line  AB,  then  the  projected  point  must 
lie  strictly  between  A  and  B  on  this  line  segment.  Also  point  C  is  selected 
a  distance  c  from  the  given  border  segment,  and  will  be  chosen  so  that  it 
satisfies  all  the  inequalities  defining  domain  D  except  for  the  inequality 


being  tested. 


Ic  must  be  shown  Chat  cest  points  selected  In  this  way  will  reliably 
detect  domain  errors  due  to  boundary  shifts.  If  any  of  the  test  points  lead 
to  an  incorrect  output,  then  clearly  there  Is  an  error.  On  the  other  hand. 

If  the  outputs  of  all  these  points  are  correct,  then  either  the  given  border 
Is  correct  or  we  have  gained  considerable  Information  as  to  the  location  of  a 
correct  herder.  Figure  2  shows  that  the  correct  border  must  lie  on  or  above 
points  A  and  B,  and  must  lie  below  point  C,  for  by  assumptions  (1)  and  (4), 
each  of  these  test  points  must  lie  in  Its  assumed  domain.  So  if  the  given 
border  Is  incorrect,  the  correct  border  can  only  belong  to  a  class  of  line 

segments  which  Intersect  both  closed  line  segments  AC  and  3C. 

Figure  2  Indicates  s  specific  correct  border  from  this  class  which 

Intersects  line  segments  AC  and  BC  at  P  and  Q  respectively.  Define  the 
domain  error  magnitude  for  this  correct  border  to  be  the  maximum  of  the  dlstanc 
from  P  and  from  Q  to  the  given  border.  Then  It  Is  clear  that  the  chosen 
test  points  have  detected  domain  errors  due  to  border  shifts  except  for  a 

class  of  domain  errors  of  magnitude  less  than  c.  In  a  continuous  space  c 
can  be  chosen  arbitrarily  small,  and  as  c  approaches  zero,  the  line  segments 
AC  snd  BC  become  arbitrarily  close  to  the  given  border,  and  in  the  limit,  we 
can  conclude  that  the  given  border  Is  Identical  to  the  correct  border.  However 
the  continuity  of  the  space  also  Implies  that  regardless  of  how  small  c  la 
chosen,  border  shifts  of  magnitude  less  than  c  may  not  be  detected,  and  there¬ 
fore  we  must  correspondingly  qualify  our  results. 

Figure  3  shows  the  three  general  types  of  border  shifts,  and  will 
u*  ***  how  the  ON-OFF-ON  sequence  of  test  points  works  In  each 
c*,e*  figure  3(a),  the  border  shift  has  effectively  reduced  domain  D^. 

Test  points  A  and  B  yield  correct  outputs,  for  they  remain  In  the  correct 
domain  Dj  despite  the  shifted  border.  However,  the  border  has  shifted  past 
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cast  point  C,  causing  It  to  be  In  domain  Instead  of  domain  .  Since 
the  program  will  now  follow  the  wrong  path  when  executing  input  C, 

Incorrect  results  will  be  produced.  In  Figure  3(b),  the  domain  has 
been  enlsrged  due  to  the  border  shift.  Here  test  point  C  will  be  processed 
correctly  since  it  is  still  in  domain  Dj,  but  both  A  and  B  will  detect  the 
shift  since  they  should  also  be  in  domain  D^.  Finally  in  Figure  3(c), 
only  test  point  B  will  be  Incorrect  since  the  border  shift  causes  it  to  be 
in  Instead  of  D^.  Therefore,  the  ON-OFF-ON  sequence  is  effective  since 
at  least  one  of  the  three  points  oust  be  in  the  wrong  domain  as  long  ss  the 
border  shift  is  of  a  magnitude  greater  than  c. 

Recall  in  Figure  2  that  we  required  the  OFF  point  C  to  satisfy  all 
the  inequalities  defining  domain  D  except  for  the  inequality  being  tested. 

The  reason  for  this  requirement  is  that  some  correct  border  segment  may 
terminate  on  the  extension  of  an  adjacent  border,  rather  than  intersecting 
both  line  segements  AC  and  BC  as  we  have  argued.  Since  we  have  assumed  a 
continuous  space,  C  could  always  be  chosen  closer  to  the  given  border  in 
order  to  satlafy  the  adjacent  border  inequalities.  An  analysis  of  this  situa¬ 
tion  trill  be  presented  in  Section  6.2. 

We  must  also  demonstrate  the  reliability  of  the  method  for  domain  errors 
in  which  the  predicate  operator  is  Incorrect.  If  the  direction  of  the 
Inequality  is  wrong,  e.g.,  _<  is  used  Instead  of  _> «  the  domains  on  either  side 
of  the  border  are  Interchanged,  and  any  point  In  either  domain  will  detect 
the  error.  A  more  subtle  error  occurs  when  Just  the  border  Itself  is  in 

the  wrong  domain,  e.g.,  _<  is  used  Instead  of  <.  In  this  case  the  only  points 
affected  lie  on  the  border,  and  since  we  always  test  ON  points,  this  type  of 
error  will  always  be  detected.  If  the  correct  predicate  is  an  equality,  the 
OFF  point  will  detect  the  error. 
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The  domain  testing  strategy  requires  at  most  3*P  test  points  for  a 
domain,  where  P,  the  number  of  border  segments  on  this  boundary,  is  bounded 
by  the  number  of  predicates  encountered  on  the  path.  However,  we  can 
reduce  this  cost  by  sharing  test  points  between  adjacent  borders  of  the 
domain.  The  requirement  for  sharing  an  ON  point  is  that  it  is  an  extreme 
point  for  two  adjacent  borders  which  are  both  closed  or  both  open.  In  the 
example  in  Figure  4,  the  points  that  can  be  shared  are  A^ ,  A^ ,  and  A^ .  The 
number  of  ON  points  needed  to  test  the  entire  domain  boundary  can  be  reduced 
by  as  much  as  one  half,  l.e.,  the  number  of  test  points,  TP,  required  to 
test  the  complete  domain  boundary  lies  in  the  following  range: 

2*P  <_  TP  <  3*P. 

Even  more  significant  savings  are  possible  by  sharing  the  test  points 
for  a  common  border  between  two  adjacent  domains.  If  both  domains  are 
tested  independently,  the  common  border  between  them  is  tested  twice,  using 
a  total  of  six  test  points.  If  this  border  has  shifted,  both  domains  must 
be  affected,  and  the  error  will  be  detected  by  testing  either  domain. 
Therefore,  the  second  set  of  test  points  can  safely  be  omitted.  However, 
the  cost  savings  in  such  sharing  should  be  balanced  against  the  additional 
processing  required. 

We  now  formally  summarize  the  results  of  this  section  in  the  following 
proposition. 

Proposition  2 

Given  assumptions  (1)  through  (8),  with  the  OFF  test  point  chosen  a 
distance  c  from  the  corresponding  border,  the  domain  testing  strategy  is 

guaranteed  to  detect  all  domain  errors  of  magnitude  greater  than  c.  More¬ 
over,  the  cost  is  no  more  than  3*P  test  points  per  domain,  where  P  is  the 
number  of  predicates  along  the  corresponding  path. 
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N-Dlmenslonal  Linear  Inequalities 

The  domain  testing  strategy  developed  for  the  two-dimensional  case  can 
be  extended  to  the  general  N-dlmenslonal  case  In  a  straightforward  manner. 

The  central  property  used  In  the  previous  analysis  was  the  fact  that  a 
line  is  uniquely  determined  by  two  points.  We  can  easily  generalize  this 
property  since  an  N-dlmenslonal  hyperplane  is  determined  by  N  linearly 
Independent  points.  So,  whereas  in  the  two-dimensional  case  we  had  to 
Identify  only  two  points  on  the  correct  border,  in  general  we  have  to  identify 
N  points  on  the  correct  border,  and  in  addition,  these  points  must  be  guaranteed 
to  be  linearly  independent. 

The  validation  of  domain  testing  for  the  general  linear  case  is  based  on 
the  same  geometric  arguments  used  in  the  two-dimensional  case.  The  key  to  the 
methodology  Is  that  the  correct  border  must  intersect  every  OFF-ON  line  segment, 
assuming  that  the  test  points  are  all  correct.  Since  we  must  identify  a  total 
of  N  points  on  the  correct  border,  we  need  N  OFF-ON  line  segments,  and  we  can 
achieve  this  by  testing  N  linearly  Independent  ON  test  points  on  the  given 
border  and  a  single  OFF  test  point  whose  projection  on  the  given  border  is  a 
convex  combination  of  these  N  points.  In  addition,  as  in  the  two-dimensional 
case,  the  OFF  point  must  also  satisfy  the  Inequality  constraints  corresponding 
to  all  adjacent  borders. 

Even  though  we  do  not  know  these  specific  points  at  which  the  correct  border 
Intersects  the  ON-OFF  segments,  we  do  know  that  these  points  must  be  linearly 
Independent  since  the  ON  points  are  linearly  independent.  The  OFF  point  Is 
a  distance  c  from  the  given  border,  and  In  the  limit  as  c  approaches  zero, 
each  OFF-ON  line  segment  becomes  arbitrarily  close  to  the  given  border. 

However,  as  in  the  two-dimensional  case,  the  c-llmltatlon  means  that  only 
border  shifts  of  magnitude  greater  than e  will  be  detected. 
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The  domain  testing  strategy  requires  at  most  (N+1)*P  test  points  per 
domain,  where  N  is  the  dimensionality  of  the  input  space  in  which  the  domain 
Is  defined  and  P  is  the  number  of  border  segments  in  the  boundary  of  the 
specific  domain.  However,  we  again  can  reduce  this  testing  cost  by  using 
extreme  points  as  ON  test  points.  Each  extreme  point  is  formed  by  the 
Intersection  of  at  least  N  border  segments,  and  therefore  one  point  can  be 
used  to  test  up  to  N  borders.  In  addition,  extreme  points  are  also  linearly 
independent.  Each  border  must  be  tested  by  N  ON  points,  and  any  points 
beyond  this  are  redundant,  and  so  not  all  extreme  points  on  each  border  are 
required.  As  a  result  of  this  kind  of  sharing,  the  number  of  test  points  can 
be  as  few  as  2*P.  As  in  the  two-dimensional  case,  there  can  be  further 
savings  if  test  points  are  shared  between  adjacent  domains.  Finally,  since 
some  of  the  P  border  segments  may  be  produced  by  the  mln-max  constraints  which 
define  the  bounds  of  the  input  space,  the  number  of  test  points  can  be 
reduced  still  further,  if  we  can  assume  that  these  constraints  are  prede¬ 
termined  and  need  not  be  tested. 

This  generallzat ion  to  N  dimensions  is  significant  since  very  few 
nontrivial  programs  have  only  two  input  variables.  We  summarize  the  results 
so  far  in  the  following  proposition: 

Proposition  3 

Given  assumptions  (1)  -  (7),  with  the  OFF  test  point  chosen  a  distance  e 
from  the  corresponding  border,  the  domain  testing  strategy  is  guaranteed  to 
detect  all  domain  errors  of  magnitude  greater  than  e  regardless  of  the  dimen¬ 
sionality  of  the  input  space.  Moreover,  the  cost  is  not  more  than  (N+1)*P 


test  points  per  domain. 
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4  .  J  Equality  and  Nonequallty  Predicates 

Equality  predicates  constrain  the  domain  to  lie  In  a  lower  dimensional 
space.  If  we  have  an  N-diaensional  Input  space  and  the  domain  is  constrained 
by  L  independent  equalities,  the  remaining  inequality  and  nonequallty 
predicates  then  define  the  domain  within  the  (N-L) -dimensional  subspace 
defined  by  the  set  of  equality  predicates. 

In  Figure  5  we  see  the  equality  border  and  the  proposed  set  of  test  points. 
In  a  general  N-dimensional  domain,  let  us  first  consider  a  total  of  N  ON 
points  on  the  border  and  two  OFF  points,  one  on  either  side  of  the  border. 

As  before,  the  ON  points  must  be  independent,  and  the  projection  of  each  OFF 
point  on  the  border  must  be  a  convex  combination  of  the  ON  points. 

Given  an  incorrect  equality  predicate,  the  error  could  be  either  in  the 
relational  operator  or  in  the  position  of  the  border  or  both.  The  proposed 
set  of  test  points  can  be  shown  to  detect  an  operator  error  or  a  position 

error  by  arguments  analogous  to  those  previously  given.  This  set  of  points 
is  also  adequate  for  almost  all  combinations  of  operator  and  position  errors, 
except  for  the  following  pathological  possibility.  Let  us  assume  that  the 
border  has  shifted  and  the  correct  predicate  is  a  nonequallty.  If  both  OFF 
points  happen  to  lie  on  the  correct  border  while  none  of  the  ON  points 
belong  to  this  border,  the  error  would  go  undetected.  This  singular 
situation  is  diagrammed  as  the  dashed  border  in  Figure  6,  where  and  are 
the  ON  points,  and  and  Cj  are  the  OFF  points.  This  problem  can  be  solved 
by  testing  one  additional  point  selected  so  that  it  lies  both  on  the  given 
border  and  the  correct  border  for  this  case,  i.e.,  at  the  Intersection  point 
of  the  given  border  with  the  line  segment  connecting  the  two  OFF  points. 

This  additional  point  is  denoted  by  B  in  the  figure. 

Each  equality  predicate  can  thus  be  completely  tested  using  a  total  of 
(N+3)  test  points.  By  sharing  test  points  between  all  the  equality  predicates. 
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this  number  can  be  considerably  reduced,  but  the  reduction  depends  upon 
values  of  N  and  L.  In  addition,  since  testing  the  equality  predicates 
reduces  the  effective  dimensionality  to  (N-L)  for  each  of  the  Inequality  and 
nonequality  borders,  and  the  equalityON  test  points  can  be  shared,  even 
further  reductions  are  possible. 

For  the  case  of  a  nonequality  border,  the  testing  strategy  is  Identical 
to  that  of  the  equality  border  Just  discussed.  The  arguments  for  the 
validity  of  the  strategy  are  analogous  to  those  In  previous  cases.  Again  in  this 

V* 

case,  Che  pathological  possibility  discussed  in  connection  with  the 
equality  prei  icate  can  occur,  and  can  be  handled  In  the  same  way.  The  major 
difference  Is  thac  while  test  points  can  be  extensively  shared  between 
equality  and  Inequality  borders,  in  general  such  sharing  is  not  possible 
between  nonequality  and  inequality  borders.  The  following  proposition 
summarizes  the  situation  for  testing  linear  borders  in  N-d imensions . 

Proposition  4 

Given  assumptions  (1)  through  (6),  with  each  OFF  point  chosen  a  distance 
t  from  the  corresponding  border,  the  domain  testing  strategy  is  guaranteed  to 
detect  all  domain  errors  of  magnitude  greater  than  c  using  no  more  than 
P*(N+3)  test  points  per  domain. 

4 . 4  An  Example  of  Error  Detection  Us  the  Domain  Strategy 

The  domain  testing  strategy  has  been  described  and  validated  using  some¬ 
what  complicated  algebraic  and  geometric  arguments.  In  this  section  we  hope  to 
complement  those  discussions  by  demonstrating  how  a  set  of  domain  test  points 
for  a  short  sample  program  actually  detects  specific  examples  of  different 
types  of  programming  errors.  In  discussing  each  error  we  will  focus  on  a 
specific  domain  affected  by  the  error,  and  a  careful  analysis  of  its  effect  on 
the  domain  will  allow  us  to  identify  those  domain  test  points  which  detect  the 


error . 
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The  short  example  program  reads  two  values,  I  and  J,  and  produces  a  single 
output  value  M.  Therefore,  the  Input  space  Is  two-dimensional,  and  the  following 
mln-max  constraints  have  been  chosen  so  that  the  Input  space  diagram  would 
not  be  too  large  or  complicated. 

-8  <  I  <  8  -5  <  J  <  5. 

In  addition,  since  this  Is  a  two-dimensional  space,  we  will  also  test  extreme 
points  for  the  border  segments  produced  by  the  mln-max  constraints  In  order  to 
be  able  to  detect  as  many  missing  path  errors  as  possible. 

Even  though  the  input  space  Is  assumed  to  be  continuous,  the  coordinates 
of  each  test  point  are  specified  to  an  accuracy  of  0.2  In  order  to  simplify  the 
diagrams  and  discussions.  Of  course.  In  an  actual  Implementation  each  OFF 
point  would  be  chosen  much  closer  to  the  border. 

The  sample  program  is  listed  below,  and  it  consists  of  three  simple 
IF  constructs,  the  first  two  of  which  are  Inequalities  and  the  last  of  which 
is  an  equality.  The  input  space  structure  is  diagrammed  in  Figure  7,  where  the 
solid  diagonal  border  across  the  entire  space  is  produced  by  the  first  predicate, 
the  dashed  horizontal  border  and  short  vertical  border  at  I»0  are  produced  by  the 
second  predicate,  and  the  vertical  equality  border  at  1*5  corresponds  to  the 
third  predicate.  In  addition,  domain  test  points  have  been  indicated  for  the 
two  domains  which  we  will  dlacuas,  viz.,  TTE  a. id  ETT. 

Statement 

Number 

READ  I.J; 

1  IF  I  <  J  +  1 

2  THEN  K  -  I  ♦  J  -  1; 

3  ELSE  K  •  2*1  ♦  1; 

ENDIF; 

4  IF  K  >  I  ♦  1 

5  THEN  L  -  I  ♦  1; 

6  ELSE  L  -  J  -  1; 

ENDIF; 

7  IF  I  -  5 

8  THEN  M  -  2*L  ♦  K; 

9  ELSE  M  •  L  ♦  2*K  -  1; 

ENDIF; 


WRITE  M; 


TTE 
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Table  II  illustrates  two  types  of  errors  we  would  like  to  consider. 

The  first  is  an  error  in  the  inequality  predicate  in  statement  #4  of  the 
above  program,  (K  >_  1+1) ,  where  it  is  assumed  that  the  correct  predicate  should 
be  (K  ^  1+2).  This  corresponds  to  an  inequality  border  shift,  and  the  modified 
domain  structure  is  shown  in  Figure  8.  Three  points  have  been  selected  to 
test  this  border,  and  it  can  be  seen  in  Table  II  that  the  two  ON  points  detect 
this  error,  where  M  and  M'  represent  the  output  variables  for  the  given  program 
and  for  the  assumed  correct  program  respectively.  Note  that  as  a  result  of 
this  error,  the  vertical  border  at  I“0  in  Figure  7  has  also  shifted  to  1-1  in 
Figure  8,  and  if  tested,  would  also  reveal  this  error. 

Table  II  also  shows  the  effect  of  an  error  in  an  equality  predicate  in 
statement  #7  of  the  given  program.  It  is  assumed  that  the  correct  predicate 
should  be  (I-5-J)  rather  than  the  (1*5)  predicate  which  occurs  in  the  given 
program.  Figure  9  shows  the  modified  input  space  structure,  and  it  can  be  seen 

that  equality  borders  TTT  and  ETT  have  shifted.  Table  II  shows  the  five  points 

which  test  the  ETT  border,  and  note  that  two  ON  points  both  detect  this  shift. 

Table  III  indicates  that  the  domain  strategy  can  alao  detect  a  compu¬ 
tation  error  and  a  missing  path  error,  even  though  we  have  previously  noted 
that  reliability  cannot  be  proven  for  these  cases.  The  computation  error 

arises  from  statement  #6  in  the  given  program,  where  it  is  assumed  that  the 
correct  assignment  statement  for  this  ELSE  clause  is  (L-I-2)  Instead  of  (L«J-1) 
which  actually  appears  in  the  given  program.  Since  L  is  not  used  in  any  sub¬ 
sequent  predicate,  this  corresponds  to  a  computation  error  rather  than  a  domain 

error.  Thus  the  input  space  structure  In  Figure  7  is  applicable  for  both 
the  given  and  the  correct  programs.  Table  III  shows  the  six  test  points  which 
have  been  chosen  to  test  domain  TEE  which  Is  affected  by  this  computation  error. 
Four  of  the  points  should  Indicate  the  error,  but  note  the  test  results  at 
(-4,  -5)  are  coincidentally  correct;  the  remaining  three  points  detect  the  error. 


FIGURE  8  Correct  Input  Space  for  a  Domain  Error 


FIGURE  9  Correct  Input  Space  for  an  Equality  Predicate  Erros 


Test  Points 

Do— In  In  Error  Clvsn  State— nt  In  Error  Aasu— d  Correct  Statens nt  for  this  Do— In 

'  . . .  . Point  H 


Tabic  III  Detection  of  a  Computation  Error 
and  Hissing  Path  Error 
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Suppose  in  program  stataaane  #2  the  THEN  clause  la  replaced  by  the 
following  code. 

THEN  IF  2*J  <  -5*1  -  40 
THEN  K  -  3; 

ELSE  K  -  I  +  J  -  1; 

ENDIF; 

This  corresponds  to  a  missing  path  error  and  is  indicated  as  such  in  Table  III. 
Figure  10  shows  how  the  domain  TEE  is  modified  by  this  missing  path  error,  but 
note  that  only  teat  point  (-8,-5)  detects  this  error.  If  the  <  Inequality  in 
the  missing  predicate  had  been  an  equality,  this  would  have  produced  a  missing 
path  error  of  reduced  dimensionality,  corresponding  to  a  domain  consisting  of 
Just  the  line  segment  in  Figure  10,  and  would  have  gone  undetected. 


FIGURE  10  Correct  Input  Space  for  a  Hissing  Path  Error 
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CHAPTER  5 


EXTENSIONS  OF  THE  DOMAIN  TESTINC  STRATEGY 

Many  assumptions  were  required  in  presenting  the  previous  results,  but 

to  some  extent  these  assumptions  were  made  to  allow  a  simple  exposition  of  the 
domain  testing  strategy.  This  section  will  discuss  assumptions  (3),  (4),  and  (5) 
which  deal  with  compound  predicates,  adjacent  domains  which  compute  the  same 
function,  and  nonlinear  borders,  respectively.  The  treatment  of  these  cases  will 
certainly  require  additional  test  points,  and  in  some  Instances  will  demand  extra 
processing  which  may  render  this  testing  approach  Impractical.  However,  one  of 
the  main  objectives  of  this  section  Is  to  Illustrate  that  none  of  the  assumptions 
(3),  (4),  or  (5)  pose  a  theoretical  limitation  to  the  domain  testing  strategy  which 
cannot  be  dealt  with  in  some  fashion. 

The  General  Nonlinear  Case 

A  finite  domain  testing  strategy  cannot  be  effective  for  the  universal  class 
of  nonlinear  borders,  but  we  must  determine  whether  this  Is  caused  by  some  funda¬ 
mental  difference  between  linear  and  nonlinear  functions.  If  the  problem  Is  that 
we  are  considering  too  general  a  class  of  borders,  then  we  should  be  able  to  extend 
the  methodology  to  cover  well-defined  subclasses  of  nonlinear  functions.  However, 
if  the  problem  is  caused  by  some  basic  characteristic  of  nonlinear  borders,  we 
will  not  be  able  to  extend  domain  testing  to  any  class  of  nonlinear  functions. 

For  linear  borders,  we  have  assumed  that  if  the  given  border  la  linear,  and 
if  there  Is  a  domain  error,  then  the  correct  border  Is  also  linear.  In  order  to 
extend  our  testing  results  to  particular  subclasses  of  nonlinear  functions,  such 
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quadratic  or  cubic  polynomials,  v«  must  assume  Chat  if  the  given  nonlinear 
border  is  in  error,  then  the  correct  border  is  in  the  same  nonlinear  class.  This 
nonlinear  class  will  be  specified  by  K  parameters;  for  example,  consider  the  general 
form  of  a  two-dimensional  quadratic  in  terms  of  variables  X  and  Y,  where  A,  B,  C,... 
are  coefficients,  and  K  •  6: 

AX2  ♦  BY2  +  CXY  +  DX  +  EY  +  F  -  0. 

Then  (K-l)  points  can  be  chosen  in  order  to  solve  for  these  K  coefficients.  For 

the  example  above,  the  five  points  [X^  Yj.  i  -  1 . 5.  should  satisfy  the  following 

system  of  equations: 


Define  an  Independent  set  of  (K-l)  points  [X  ,  Y^  as  a  set  which  can  be  used  to 
solve  for  the  coefficients,  and  thus  determine  a  specific  member  of  the  nonlinear 

class. 

He  can  now  formulate  the  general  nonlinear  domain  testing  strategy  in  terms 
of  these  observations.  (K-l)  ON-OFF  pairs  of  points  are  chosen  such  that  the 
(K-l)  ON  points  are  Independent  and  each  OFF  point  is  chosen  a  distance  c  from  the 
corresponding  ON  point.  This  requires  2*(K-1)  teat  points  per  nonlinear  border. 

The  (K-l)  ON-OFF  line  segments  formed  by  this  set  of  pairs  have  been  chosen  so  that 
the  only  correct  borders  which  yield  correct  test  results  must  Intersect  each  of 
these  ON-OFF  line  segments.  For  any  particular  correct  border,  there  are  (K-l) 
Independent  Intersection  points,  which  determines  the  border  completely.  Note  that 
the  Intersection  points  are  Independent  if  c  is  chosen  sufficiently  small,  since 
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Che  ON  poincs  are  independent  for  the  given  border.  A  further  requirement,  as  in 
the  linear  case,  is  that  all  OFF  points  satisfy  all  Inequality  borders  other  than 
Che  one  being  tested. 

While  a  single  OFF  point  was  sufficient  in  the  linear  case,  the  Independence 
criterion  requires  (K-l)  OFF  points  for  each  nonlinear  border.  In  the  former  case 
linearity  allowed  the  OFF  point  to  be  shared  by  all  the  ON  points,  since  the  linear 
independence  of  the  points  identified  as  lying  on  the  true  border  is  guaranteed 
by  the  linear  independence  of  the  ON  points  themselves.  If  we  were  to  test  a  non¬ 
linear  border  with  (K-l)  ON  points  and  a  single  OFF  point,  we  would  be  able  to 
conclude  that  the  correct  and  given  borders  intersect  at  (K-l)  points.  However, 
we  cannot  conclude  chat  these  (K-l)  points  are  independent.  We  know  of  no 
selection  criterion  for  the  ON  points  which  would  guarantee  the  independence 

of  the  intersection  points  using  only  one  OFF  point.  So  an  effective  strategy 
requires  the  full  set  cf  2*K  test  points,  and  unfortunately  K  grows  very  rapidly 
as  the  dimensionality  and  degree  of  nonlinearity  of  the  border  Increases. 

A  two-dimensional  nonlinear  border  is  a  very  special  case,  and  even  chough 
the  general  strategy  is  effective,  a  slightly  different  testing  strategy  can  be 
formulated  to  reduce  the  number  of  required  test  points.  The  basic  difference  is 

that  the  Intersection  between  two-dimensional  nonlinear  borders  from  the  same 
class  is  a  finite  set  of  points,  the  maximum  number  of  which  can  be  determined 
from  the  form  of  the  function.  For  example  a  pair  of  two-dimensional  quadratic 
curves  can  intersect  In  at  most  four  points.  This  means  that  any  set  of  more  than 
four  points  cannot  possibly  lie  on  two  distinct  quadratics,  and  any  five  points 
uniquely  determines  a  specific  quadratic.  Therefore,  we  do  not  have  to  worry 
about  independence  In  the  two-dimensional  case,  since  any  set  of  (K-l)  distinct 
points  will  produce  a  system  of  Independent  linear  equations.  For  example, 
any  three  distinct  points  can  lie  on  at  most  one  circle,  since  two  circles 
cannot  have  more  than  two  points  in  common. 
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We  test  a  two-dimensional  nonlinear  border  with  K  points,  e.g.,  six  for 
a  quadratic  selected  in  an  ON-OFF-ON-OFF. . . .  sequence  along  the  border  as  diagrammed 
for  the  closed  border  in  Figure  11 .  Since  the  correct  border  must  pass  on  or 
above  the  given  border  at  each  ON  point,  and  must  pass  below  each  OFF  point,  the  two 
borders  must  intersect  an  odd  number  of  times,  let  us  assume  once,  in  each  ON-OFF  and 
OFF -ON  interval  along  the  border.  The  K  test  points  define  (K-l)  intervals  on 
the  border,  each  of  which  must  contain  at  least  one  Intersection  point.  We  have 
shown  that  these  (K-l)  points  must  be  independent,  and  since  they  cannot  lie 
on  two  distinct  borders,  the  given  border  must  be  correct  within  c.  As  a 
technical  detail,  it  is  also  possible  that  the  correct  border  may  be  tangent  to 
the  given  border  at  an  ON  point,  but  if  this  occurs,  an  argument  involving  the 
derivatives  of  the  two  borders  at  that  point  can  be  invoked  to  Justify  tne  choice 
of  the  test  points  for  this  two-dimensional  case. 

Although  the  domain  strategy  has  been  extended  to  nonlinear  boundaries, 
points  must  be  generated  in  a  domain  defined  by  nonlinear  boundaries,  requiring 
the  solution  of  nonlinear  systems  of  equations.  Since  this  probably  requires 
excessive  processing  for  arbitrary  nonlinear  borders,  it  does  not  represent  a 
very  practical  approach. 

Adjacent  Domains  Which  Compute  the  Same  Function 

If  two  adjacent  domains  compute  the  same  function,  any  test  point  selected 
for  their  cotmnon  border  is  Ineffective,  since  the  same  output  values  are 
computed  for  the  test  ooint  regardless  of  the  domain  in  which  it  lies.  We 
will  demonstrate  how  domain  testing  can  be  modified  to  deal  with  this  problem. 

In  Figure  12(a),  assuming  domain  D,  were  being  tested,  we  must  compare 
the  functions  calculated  in  domains  and  D.,  for  test  point  A,  and  for 
B,  and  and  for  C.  One  of  the  major  problems  to  be  solved  is  the 
identification  of  these  adjacent  domains.  We  assume  that  when  testing  domain 
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Dj  the  partitioning  structure  of  the  adjacent  domains  and  the  program  paths 
associated  with  these  domains  Is  not  known.  It  would  be  very  complicated 
to  have  to  generate  the  domains  which  are  adjacent  to  the  border  being  tested. 

Figure  12(b)  Illustrates  an  approach  to  this  problem.  The  border  being 
tested  Is  shifted  parallel  by  a  small  distance  c,  so  that  test  points  A  and  B 
now  belong  to  adjacent  domains,  D,  and  ,  respectively.  The  modified  program 
Is  then  retested  using  test  points  A  and  B,  which  will  as  a  by-product  Identify 
the  paths  associated  with  these  two  adjacent  domains.  Ue  can  then  compare 
the  output  for  each  test  point  before  and  after  the  shift.  If  It  is  different, 
then  we  can  definitely  conclude  that  the  adjacent  domain  computes  a  different 
function,  and  this  test  point  can  safely  be  used.  If  the  output  Is  the  same 
for  that  test  point,  then  we  can  conclude  that  either  assumption  (1) 
or  (4)  Is  violated.  However,  there  is  no  way  to  decide  this,  and  the  only 
practical  approach  is  to  use  further  test  points.  If  we  know  that  coincidental 
correctness  cannot  occur,  then  we  could  conclude  on  the  basis  of  a  single  point 
that  the  adjacent  domain  computes  the  same  function. 

If  two  adjacent  domains  such  as  and  D,  In  Figure  12(a)  are  found  to 

compute  the  same  function,  then  In  order  to  carry  out  the  domain  testing  strategy 

on  the  given  border,  new  test  points  may  have  to  be  selected.  For  example, 
point  A  can  no  longer  be  used,  and  this  requires  ascertaining  the  border  structure 

between  Dj  and  D,.  Thus  a  considerably  amount  of  processing  is  required  which 

Is  probably  not  practical. 

In  summary,  a  technique  of  testing  each  point  twice  will  assure  us  that 
assumption  (4)  Is  valid,  and  this  redundancy  might  be  viewed  as  a  reasonable  price 
to  pay  to  eliminate  this  restriction.  However,  if  an  Instance  is  found  where  the 
assumption  is  not  valid,  a  basic  theoretical  problem  exists. 
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Domain  Testing  for  Compound  Predicates 

Assumption  (3)  stated  that  a  path  contained  only  simple  predicates,  and 
this  Implied  that  the  set  of  Input  points  could  be  characterized  quite 
simply  as  a  single  domain.  We  must  consider  what  complications  can  occur 
for  compound  predicates,  and  how  the  domain  strategy  can  be  generalized 
to  test  paths  containing  these  predicates. 

The  set  of  Inputs  corresponding  to  a  path  Is  defined  by  the  path 
condition,  consisting  of  the  conjunction  of  the  predicates  encountered  along 
the  path.  If  a  compound  predicate  of  the  form  (C(l)  AND  C(1+1)J  la  encountered 
on  the  path,  the  path  condition  Is  still  a  single  conjunction  of  simple 
predicates,  and  the  only  difference  Is  that  two  of  the  simple  predicates 
are  produced  as  a  single  branch  point  on  the  path.  No  modifications  of  the 
domain  testing  strategy  are  required  in  this  case. 

However,  compound  predicates  using  the  Boolean  operator  OR  are  more 
complicated.  Consider  a  path  containing  the  following  predicates: 

Cl*  C2 . *C1  °R  Cl+J’  "•  Ct 

The  path  condition  In  this  case  Is  the  conjunction  of  these  predicates,  and 
In  standard  disjunctive  normal  form: 


(C.  AND  C,  AND  ...  ANT  C.  AND  ...  AND  C„ ] 
l  i  X  t 

OR  (Ct  AND  C,  AND  ...  AND  C  ^  AND  ...  AND  Cf 1 

The  set  of  Input  data  points  following  this  path  consists  of  the  union  of  two 
domains,  each  defined  by  the  conjunction  of  simple  predicates,  and  in  general 


any  number  of  these  domains  are  possible. 

Assuming  linear  predicates,  each  of  these  domains  Is  a  convex  polyhedron, 
but  the  domains  may  overlap  In  arbitrary  ways.  The  major  problem  caused  by 
these  compound  predicates  Is  that  the  domains  correspond  to  the  same  path,  and 
the  assumption  that  adjacent  domains  do  not  compute  the  same  function  Is  violated. 
We  identify  three  cases  of  Importance:  domains  which  do  not  overlap,  domains 

which  partially  overlap,  and  domains  which  totally  overlap. 
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The  first  case  is  Indicated  In  Figure  13(a),  where  domains  Dj  and  D., 
are  defined  by  the  compound  predicate  [C^  OR  C,],  and  domain  corresponds  to 

some  other  path.  In  this  case  our  methodology  can  be  applied  to  each  domain 
separately,  since  the  two  domains  for  this  path  are  not  adjacent. 

In  Figure  13(b),  the  domains  partially  overlap,  where  U  D.,  Is  the 
domain  defined  by  C^,  and  U  D ^  Is  the  domain  defined  by  C.,.  In  the  example 
we  cannot  test  the  domains  separately,  since  they  are  adjacent  and  the  same 
function  Is  computed  in  each.  For  example,  any  test  point  for  C^,  selected 
along  that  part  of  the  border  between  and  Dj,  Is  Ineffective  since  the  same 
results  are  computed  for  It  In  both  of  these  regions.  So,  in  this  case  we 
must  Insure  that  the  adjacent  domain  assumption  Is  satisfied  by  selecting 
test  points  for  Cj  and  C.,  which  lie  In  that  part  of  the  border  adjacent  to  a 
domain  for  some  other  path. 

In  order  to  deal  effectively  with  this  case,  some  extra  analysis  will  have  to 
be  made,  first  In  order  to  identify  this  second  case,  and  also  to  Identify  the 
actual  domain,  which  Is  no  longer  convex.  The  borders  of  this  domain  are  shown 
In  bold  face  in  Figure  13(b).  This  Is  probably  no  longer  a  practical  approach, 
especially  for  higher  dimensions. 

The  third  case  Is  shown  in  Figure  13(c),  where  the  domain  D,  for  predicate 
Is  a  subset  of  the  other  domain,  U  D ,  which  Is  obtained  for  predicate  C^. 
This  presents  a  serious  problem  since  there  are  no  test  points  for  border  B  of 
domain  which  can  satisfy  the  adjacent  domain  assumption,  and  therefore  B  cannot 
be  tested  effectively.  The  technique  developed  In  the  previous  section  should 
help  to  Identify  this  case.  However,  even  If  this  case  could  be  Identified,  testing 
for  border  B  le  no  longer  a  practical  procedure. 

So,  In  susnary,  a  compound  predicate  of  the  form  (Cl  AND  C2]  le  the  same  as 
two  simple  predicates,  and  domain  testing  can  be  applied  to  a  domain  defined  with 
this  type  of  compound  predicate.  In  addition.  If  the  compound  predicate 
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is  of  Che  form  [C^  OR  C, J  and  Che  domains  are  distinct,  domain  testing  can  be 
applied  to  each  domain  separately.  However,  If  the  domains  overlap,  this 
Introduces  the  problem  of  adjacent  domains  which  compute  Che  same  function. 
Although  we  may  not  find  effective  test  points  for  domains  which  overlap  in 
arbitrary  ways,  we  can  recognize  this  situation  and  Identify  It  as  a  border 
which  cannot  be  tested  effectively. 
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CHAPTER  6 

ERROR  ANALYSIS  OF  DOMAIN  BORDERS  AND  DISCRETE  SPACES 

An  error  analysis  of  domain  borders  is  needed  Co  resolve  che  following 
questions : 

1)  How  smell  should  c  be  chosen  In  selecting  an  OFF  test  point  for  llneer 
borders,  end  where  are  optimal  locations  for  the  test  points? 

We  required  the  OFF  test  point  for  a  given  border  to  satisfy  all  In¬ 
equality  borders  except  that  being  tested;  how  do  potential  errors 
In  other  borders  of  the  domain  affect  this  requirement? 
ill)  What  are  the  difficulties  In  applying  domain  testing  In  a  discrete 

space  or  In  a  space  In  which  numerical  values  can  only  be  represented 
with  finite  resolution,  and  can  these  difficulties  be  circumvented  by 
taking  reasonable  precautions  with  the  method? 

These  and  other  error  analysis  problems  are  dealt  with  in  detail  In 
reference  [ij.  It  is  Interesting  to  observe  that  the  answers  to  questions 
1),  11),  and  111)  all  Involve  the  same  worst-case  situation:  when  two  adjacent 
linear  borders  of  the  same  domain  are  nearly  parallel.  Figure  14  indicates 
Che  two  cases  which  can  arise  from  adjacent  linear  borders  which  are 
nearly  parallel.  Figure  14(a)  shows  a  given  border  segment  EF  In  which  the 
two  adjacent  border  segments  EP  and  FQ  both  make  large  external  angles 

•nd  ®2*  n**r  180* .  with  the  given  border  EF.  This  leads  to  very  small 
supplementary  Internal  angles  and  t2,  and  especially  for  l2%  this  results 
In  a  very  sharp  "corner"  of  the  domain.  In  Figure  14(b),  the  adjacent  borders 
PE  and  FQ  are  again  nearly  parallel  to  the  given  border  EF,  but  a  different 
case  Is  created.  In  this  case,  external  anglAs  ^  and  0,  are  very  small, 
and  che  Internal  angles  and  #2  are  both  near  ISO*. 
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its  Which  are  Nearly  Parallel 


We  will  briefly  argue  In  chls  report  that  one  of  these  two  situations  Is 
the  key  to  the  analysis  of  questions  1),  11),  and  111),  and  we  refer  the 
reader  to  reference  [12]  for  further  details  and  proofs.  Section  6.1  Intro¬ 
duces  an  error  neasure  which  will  Indicate  the  best  location  for  each  of  the 
three  test  points.  Section  6.2  will  deal  with  the  problem  of  how  Interacting 
border  changes  may  affect  the  location  of  the  test  points.  Section  6.3  briefly 
Introduces  the  problem  of  domain  testing  In  discrete  spaces,  and  gives  a 
sufficient  condition  to  guarantee  effective  test  points  can  always  be  chosen. 

Since  all  the  above  arguments  are  given  only  for  two  dimensions.  Section  6.4 
will  show  that  the  same  basic  approach  Is  effective  for  higher  dimensions. 

An  Error  Measure  for  Test  Point  Selection 

In  Figure  14(a),  consider  the  selection  of  three  test  points  A,  B,  and  C 
for  testing  border  segment  EF.  It  Is  shown  In  reference  [12]  that  the  best 
positions  for  two  of  them,  say  A  and  B,  are  points  E  and  F,  so  the  remaining 
problem  U  the  location  of  teet  point  C.  We  have  observed  that  If  the  given 
border  EF  la  In  error,  then  test  points  A,  B  and  C  will  fall  to  detect  errors 
if  the  correct  border  Is  one  which  Intersects  line  segments  AC  and  BC.  Thus 
given  C  which  Is  at  a  distance  c  from  the  given  border  and  halfway  between 
A  and  B,  an  appropriate  error  criterion  could  be  the  "number"  of  erroneous 
points  which  would  be  undetected,  l.e.,  the  area  between  the  two  b.  rders,  possibly 
limited  by  either  or  both  of  the  extensions  of  the  adjacent  borders  EP  and  FQ. 

It  can  be  shown  that  this  area  measure  can  be  bounded  by  the  expression 

r.iM2 _ 

EF  ♦  2c  cot  0  , 


where  0  is  the  lerger  of  9j  and  02 
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In  order  for  this  error  measure  to  be  finite,  it  Is  necessary  that  both  6 \ 

£P 

and  d2  are  not  t0°  cloaa  to  180*  for  given  c.  If  |cot  0|<<  —  ,  then  the  error 
measure  is  on  the  order  of  e-EF.  This  gives  some  guidance  as  to  the  choice  of 
c  for  point  C. 

6.2  Interacting  Border  Segments 

In  presenting  the  domain  strategy,  we  required  the  OFF  test  point  to  satisfy 
all  inequality  borders  except  the  border  being  tested.  Usually  this  does  not 
impose  much  of  a  constraint  on  the  choice  of  the  OFF  point,  but  Figure  14(b) 
illustrates  a  situation  in  which  a  severe  constraint  exists.  We  can  show  that 

h _ ET 

(cot  6j  ♦  cot  92)  , 

and  since  r  <  h  for  choosing  the  OFF  test  point,  this  again  shows  the  effect  if 

either  9j  or  02  or  both  are  very  small. 

The  sane  situation  applies  for  interacting  adjacent  borders,  and  is  illus¬ 
trated  in  Figure  15.  As  long  as  the  OFF  points  and  C2  for  each  of  the  adjacent 
borders  are  chosen  sufficiently  close  to  those  borders,  and  the  external  angles 

9;  and  92  are  not  too  email,  then  the  adjacent  borders  have  a  minimal  influence 
on  the  selection  of  the  OFF  point  C  for  border  EF.  For  example,  point  C 
must  lie  inaio '  triangle  EFT'  determined  by  given  borders  EP  and  FQ.  The  correct 
borders  which  pose  the  worst  case  In  limiting  the  selection  of  point  C  are 
shown  as  dashed  lines  in  Figure  15;  these  limiting  correct  borders  are  determined 
by  how  close  Cj  and  C,  have  been  chosen  to  their  respective  test  borders.  As  a 

result  of  these  conditions,  point  C  is  constrained  to  lie  within  triangle  EFV, 

a  more  restrictive  condition  than  presented  by  triangle  EFU.  It  should  be 
clear  that  if  either  or  is  too  small,  or  either  or  C,,  Is  chosen  too 

far  fror  ft?  res-ect'.ve  tent  border,  th"  region  from  which  C  could  be  chrsen 
would  become  restrict ively  small. 
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6 . J  Discrete  Space  Analysts 

The  previous  several  sections  have  indicated  that  if  adjacent  borders  are 

nearly  parallel,  then  test  point  C  is  required  to  lie  very  close  to  the  border 

being  tested.  But  in  a  discrete  space  this  could  cause  a  severe  problem,  for  no 

discrete  point  may  exist  that  close  to  the  border.  Similar  probleas  exist  for  the 
* 

ON  test  points  A  and  B,  for  it  may  not  be  possible  to  choose  them  at  extreme 
points  of  the  border. 

For  the  discrete  space  we  shall  consider  a  two  dimensional  lattice,  with 
uniform  spacing  A  in  both  dimensions.  This  models  the  situation  where  the 
same  data  representation.  Integer  or  fixed-point,  is  used  for  two  input  variables. 

For  simplicity,  let  us  again  assume  that  points  A  and  B  can  be  chosen  as 
points  E  and  F.  We  shall  present  a  sufficient  condition  for  a  given  domain  with¬ 
in  this  discrete  lattice  which  guarantees  that  an  OFF  point  C  can  be  chosen  as 
a  lattice  point  for  each  border  so  that  the  area  criterion  of  Section  6.1  is  finite. 
The  result  is  based  upon  the  following  two  observations.  First,  any  circle  of  diameter 
A  always  contains  at  least  one  lattice  point.  Second,  from  Figure  14(a), 
note  that  if  either  external  angles  or  6i  are  too  near  180*,  then  the  "width" 
of  the  domain  will  tend  to  be  very  small  in  terms  of  the  lattice  resolution  A. 

More  formally,  define  the  d lameter  d  of  a  convex  polygonal  domain  to  be 
the  shortest  distance  from  any  extreme  point  to  any  domain  edge  not  adjacent 
to  that  extreme  point;  this  corresponds  to  our  informal  argument  about  domain 
"width".  The  sufficient  condition  can  then  be  staled  as: 


Proposition  5 
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Given  a  domain  with  diameter  d  In  a  lattice  with  resolution  A,  if 

d  >  (— )  A  -  (2.12)  A, 

n 

then  a  lattice  OFF  point  can  be  chosen  for  every  border,  and  moreover  all 
external  angles  dj  and  d2  are  constrained  by 

,  EF 

|cot  0i  +  cot  021  *  f  ,  V  • 

6?/ 


It  Is  clear  that  there  are  some  domains  in  a  discrete  space  which  cannot 
be  tested,  but  these  are  pathological  cases  where  one  of  the  domain  dimensions 
Is  on  the  order  of  the  lattice  resolution.  Moreover,  the  result  Indicates  a 
simple  computation  in  terms  of  the  domain  diameter  to  determine  when  such 
domains  are  presented  for  testing.  For  domains  which  can  be  tested  In  a  discrete 
space,  the  Important  result  from  Proposition  3  is  that  a  restriction  has  been 
obtained  on  angles  0j  and  02  which  precludes  both  angles  which  are  close  to  180° 
and  angles  which  are  too  small. 

Extensions  of  Error  Analysis  to  Higher  Dimensions 

The  previous  arguments  have  all  been  made  for  two  dimensions,  so  It  Is  Important 
that  the  essential  Ideas  can  be  generalized  to  higher  dimensions.  We  can  observe 
that  if  two  border  segments  are  adjacent,  they  are  intersecting  hyperplanes.  Again, 
problems  may  arise  if  these  two  hyperplanes  and  are  nearly  parallel,  and 
this  can  be  measured  by  taking  the  inner  product  of  their  unit  normal  vectors 
n^  and  n^,  yielding  the  cosine  of  the  angle  a  betveen  them: 


cos  a  ■  nj'n2 
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Consider  Figure  16  which  Indicates  the  testing  strategy  for  three  .J  intensions . 

H|  is  assumed  to  be  the  border  to  be  tested  by  ON  points  A  ,  A.,,  A..  and  C  is 

1  J 

the  OFF  point.  H7  is  an  adjacent  border  nearly  parallel  to  ,  and  intersects 
H,  at  line  L.  If  it  is  suspected  that  C  may  not  be  chosen  close  enough  to  H^, 
only  those  borders  which  make  an  angle  a  of  10*  or  less  with  need  to  be  in¬ 
vestigated  further. 

To  determine  a  test  point  C,  we  need  to  select  that  correct  border  hyper¬ 
plane  which  is  the  worst  case  relative  to  border  H,,  and  then  determine  whether 
or  not  these  two  hyperplanes  intersect.  This  computation  is  quite  straight¬ 
forward,  and  the  following  algorithm  together  with  Figure  16  should  indicate  how 
it  can  be  accomplished: 

(a)  select  the  ON  point  A^  furthest  from  line  L  (this  is  A^  in  Figure  16); 
the  worst  case  correct  border  hvperplane  is  then  determined  by  line 
L  and  line  segment  A^C; 

(b)  drop  a  perpendicular  line  segment  from  A^  to  line  l;  this  makes  an 
angle  6  with  line  segment  A^C',  where  C'  is  the  projection  of  point 
C  down  on  the  hyperplane  H,  being  tested;  recall  that  C'  is  known, 
for  point  C  is  obtained  by  first  finding  C'; 

(c)  the  angle  t  between  and  can  be  found  by 

c 

tan  $  "  - -  ; 

AjC'  cos  e 

(d)  if  t  <  a,  then  hyperplanes  H ,  and  do  not  intersect;  otherwise,  c 
should  be  chosen  smaller  so  that  this  condition  is  satisfied. 

Again,  in  this  analysis,  the  fact  that  adjacent  borders  H,  and  H,  are  nearly 
parallel  proves  to  be  the  key  point  in  selecting  test  point  C.  Yet,  the  above 
algorithm  can  be  used  to  choose  C  so  as  to  compensate  for  this  condition. 


Hyperplane  H 


LOOKING 
LINE  L 


F’GURE  16  Error  Analysis  In  Three  Dimensions 
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CONCLUSIONS  AND  FUTURE  WORK 

The  basic  goal  of  this  research  Is  to  replace  the  intuitive  principles 
behind  current  testing  procedures  by  a  methodology  based  on  a  formal  treatment 
of  the  program  testing  problem.  By  formulating  the  problem  in  basic  geometric 
and  algebraic  terns,  we  have  been  able  to  develop  ar.  effective  testing  methodology 
whose  capabilities  can  be  precisely  defined.  In  addition,  since  program  testing 
cannot  be  completely  effective,  we  have  identified  the  limitations  of  the  strategy. 
In  several  cases  these  limitations  have  proven  to  be  theoretical  problems  inherent 
to  the  general  program  testing  process. 

The  main  contribution  of  this  research  is  the  development  of  the  domain 
testing  strategy.  Under  certain  well-defined  conditions  the  methodology  Is 
guaranteed  to  detect  domain  domain  errors  in  linear  borders  greater  than  some  small 
magnitude  c.  Furthermore,  the  cost,  as  measured  by  the  number  of  required  test 
points,  is  reasonable  and  grows  only  linearly  with  both  the  dimensionality  of  the 
input  space  domain  and  the  number  of  path  predicates.  Domain  testing  also  detects 
transformation  errors  and  missing  path  errors  in  many  cases,  but  the  detection  of 
these  two  classes  of  errors  cannot  be  guaranteed. 

Domain  testing  has  also  been  extended  to  classes  of  nonlinear  borders,  and  we 
have  shown  that  the  methodology  generalizes  to  any  class  of  functions  which  can 
be  described  by  a  finite  number  of  parameters.  Unfortunately,  nonlinear  predicates 
pose  problems  of  extra  processing  which  probably  preclude  testing  except  for  re¬ 
stricted  cases.  For  example,  Just  finding  intersection  points  of  a  set  of  linear 
and  nonlinear  borders  can  require  an  inordinate  amount  of  processing. 

Coincidental  correctness  is  a  theoretical  limitation  inherent  to  the  program 
testing  process,  and  we  have  argued  that  it  prevents  any  reasonable  finite  testing 
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procedure  from  being  completely  reliable.  In  particular,  the  possibility  of  coin¬ 
cidental  correctness  means  that  an  exhaustive  test  of  all  points  in  an  Input 
domain  Is  theoretically  required  to  preclude  the  existence  of  computation  errors 
on  a  path.  Within  the  class  of  all  computable  functions  there  exist  functions 
which  coincide  at  an  arbitrarily  large  number  of  points,  but  if  there  is 
sufficient  resolution  in  the  output  space,  coincidental  correctness  should  be  a 
rare  occurrence  for  functions  commonly  encountered  in  data  processing  problems. 

The  class  of  missing  path  errors,  particularly  those  of  reduced  dimensionality, 
has  proven  to  be  another  theoretical  limitation  to  the  reliability  of  any  finite 
testing  strategy.  Although  our  methodology  cannot  be  guaranteed  to  detect  all 
instances  of  this  type  or  error,  it  can  be  extended  to  detect  some  well-defined 
subclasses  of  missing  path  errors.  Unfortunately,  the  extra  cost  of  this  modi¬ 
fication  may  be  unacceptably  high.  Our  analysis  of  missing  path  errors  has 

shown  that  the  cause  of  the  difficulty  is  that  the  program  does  not  contain  any 
indication  of  the  possible  existence  of  a  missing  path  error.  Therefore,  without 
additional  information,  a  reasonable  testing  strategy  for  this  class  of  errors 
cannot  be  formulated. 

The  domain  testing  strategy  requires  a  reasonable  number  of  test  points  for 
a  single  path,  but  the  total  cost  may  be  unacceptable  for  a  large  program  con¬ 
taining  an  excessive  number  of  paths.  In  particular,  this  may  occur  for  large 
programs  with  complicated  control  structures  containing  many  iteration  loops. 
Additional  research  is  needed  to  substantially  reduce  the  number  of  potential 

paths.  One  area  being  investigated  takes  advantage  of  the  fact  that  program 
modules  are  often  Independent  in  that  the  control  flow  of  one  does  not  depend 
upon  variables  defined  in  the  other.  In  this  way  the  combinatorial  growth  of  the 
number  of  domains  to  be  tested  can  be  controlled,  and  the  domain  strategy  can  be 
made  more  practical.  It  remains  to  be  shown  to  what  extent  this  Independence 
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property  can  be  applied,  and  experimental  evidence  is  needed  of  how  frequently 
independent  module*  occur  in  widely  available  programs. 

Wo  have  assumed  that  an  "oracle"  exists  which  can  always  determine  whether 
a  specific  test  case  has  been  computed  correctly  or  not.  In  reality,  the 
programmer  himself  must  make  this  determination,  and  the  time  spent  examining 
and  analyzing  these  test  cases  is  a  major  factor  in  the  high  cost  of  software 
development.  One  possible  avenue  for  future  research  would  be  to  automate  this 
process  by  using  some  form  of  input-output  specification.  If  the  user 
provides  a  formal  description  of  the  expected  results,  the  correctness  of  each 
test  case  can  be  decided  automatically  by  determining  whether  the  output 
specification  is  satisfied.  This  would  reduce  the  cost  of  testing  tre¬ 
mendously,  and  these  new  testing  techniques  would  gain  acceptance  more  quickly 
since  the  tedious  task  of  verifying  test  data  would  be  eliminated.  In 
addition,  any  extra  information  supplied  by  the  user  might  be  useful  in 
specifying  special  processing  requirements  which  would  indicate  the  existence 
of  a  possible  missing  path  error. 

The  domain  test  strategy  is  currently  being  implemented,  and  will  be 
utilized  as  an  experimental  facility  for  subsequent  research.  Experiments 
should  indicate  what  sort  of  progransning  errors  are  most  difficult  to  detect, 
and  should  yield  extensive  dynamic  testing  data.  A  most  important  contri¬ 
bution  would  be  to  indicate  both  programming  language  constructs  and  programming 
techniques  which  are  easier  to  test,  and  thus  would  produce  more  reliable  software. 
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igigswrag." 

Frofru  tasting  Is  an  inherently  practical  activity,  slnca  avary  coaiputar  program  must 
ba  tasted  before  any  confidence  can  ba  gained  that  tha  program  par  form*  its  lntandad 
function.  Soma  of  tha  bast  designed  software  has  required  that  naarly  as  auch  affort 
ba  spant  planning  and  implementing  tha  tasting  procass  as  was  lnvastad  In  tha  actual 
coding.  What  tha  practltlonar  naads  ara  battar  guidallnas  and  systematic  approachas 
In  tha  dasign  of  tha  tasting  procass  to  raplaca  tha  »4  hoc  approach  which  Is  now  so 
pravalant  in  tha  tasting  of  computer  aoftwara. 

It  would  ba  ldaal  If  thara  aalatad  a  'thaory  of  tasting'  which  could  ba  usad  to  rigor¬ 
ously  salact  program  tast  points.  Tha  problam  has  unfortunatsly  provan  so  lntractabla 
that  no  comprehensive  tatting  thaory  exists.  Rataarch  by  Goodanough  and  Garhart  (007) 
and  Howdan  (008,009)  has  rasultad  in  an  accaptad  body  of  thaory  concarnlng  tasting, 
and  has  providad  a  rigorous  basis  for  furthar  rasaarch  In  this  araa. 

Tha  objective  of  this  papar  la  to  praaant  a  mathodology  for  tha  automatic  talactlon  of 
tast  data.  Undar  sppropnata  assumptions,  this  mathodology  will  ganarata  tast  data 
which  will  datact  a  particular  class  of  arrort  in  a  program,  vis,  'domain  arrors'  as 
daflnad  by  Howdan  (009).  Tha  proposed  mathodology  Is  also  described  In  greater  detail 
In  Cohan  and  white  (OOJ)  and  In  Cohan  (004). 

Tha  goal  of  tha  tasting  procass  Is  limited  to  tha  successful  detection  of  a  program 
error  If  any  axists.  Any  attempt  to  Identify  tha  error,  its  causa,  or  an  appropriate 
correction  is  properly  categorised  as  eebwgeine.  and  Is  beyond  tha  scope  of  our  goal 
in  tha  tasting  process.  Thus  tasting  is  essentially  error  detection,  while  debugging 
is  tha  more  difficult  process  of  error  correction.  Of  course.  In  practice  these  two 
activities  usually  overlap  and  ara  frequently  combined  into  a  single  testing/debugging 
phase  in  tha  software  developawnt  cycle. 

An  important  assumption  in  our  work  Is  that  tha  user  (or  an  'oracle')  is  available, 
who  can  decide  unequivocally  if  tha  output  la  correct  for  tha  specific  input  processed. 
Tha  oracle  decides  only  If  tha  output  values  ara  correct,  and  not  whether  they  ara  cam- 
puted  correctly.  If  they  are  incorrect,  tha  oracle  does  not  provide  any  information 
about  tha  error  and  does  not  give  tha  correct  output  values. 

Tha  organisation  of  tha  papar  is  as  follows.  In  tha  first  section,  soma  preliminary 
concepts  ara  daflnad  and  discussed.  Seme  assumptions  must  ba  made  concarnlng  tha  langu¬ 
age  in  which  tha  given  computer  program  is  written,  and  tha  ramifications  of  certain 
language  constructs  are  explored.  Tha  important  concepts  of  program  path  and  path  pred¬ 
icates.  together  with  domains,  ara  daflnad  and  character  Had.  Tha  case  of  linear 
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prtdlcatti  la  given  particular  emphasis.  alnca.  In  that  situation,  tha  domains  anuaa 
tha  simple  fora  of  convex  oolyhedra  In  tha  Input  spaca. 

Log  leal  arrora  In  a  computer  program  can  ba  viewed  as  belonging  to  ona  of  two  claaaas 
of  arrorat 

a  'Domain  arrora* 
a  'Computation  arrora* , 

Informally,  a  domain  art  or  occurs  whan  a  spaclflc  input  follows  tha  wrong  path  dua  to 
an  arror  in  tha  control  flow  of  tha  program.  A  path  contains  a  coaputatlon  error  whan 
a  spaclflc  Input  follows  tha  corract  path,  but  an  arror  in  soma  assignment  stataaant 
causes  tha  wrong  function  to  ba  computed  for  ona  or  more  of  tha  output  variables. 

Tha  proposed  methodology,  called  tha  domain  strategy.  Is  designed  specifically  to  de¬ 
tect  domain  errors.  *e  will  discuss  two  fundamental  limitations  inherent  to  any  finite 
test  strategy.  Ona  such  limitation  might  ba  termed  coincidental  correctness.  This 
occurs  whan  tha  computation  for  a  spaclflc  test  point  is  Incorrect,  but  tha  output  value 
happens  to  coincide  with  tha  corract  value.  This  test  point  would  than  ba  of  no  asslst- 
*nca  In  tha  detection  of  the  domain  arror  which  caused  tha  change  in  control  flow. 
Another  inherent  tasting  limitation  has  been  identified  by  Howden  (009),  and  might  ba 
called  a  anting  pten  error,  in  which  a  required  predicate  does  not  appear  in  tha  given 
program  to  ba  tasted.  This  could  result  in  a  situation  where  no  tasting  strategy  can 
systematically  determine  that  such  a  predicate  should  ba  present. 

Tha  domain  strategy  Is  developed  by  utilising  tha  structure  of  tha  Input  spaca  corres¬ 
ponding  to  tha  program,  more  specifically,  tha  control  flow  partitions  tha  input  spaca 
Into  a  tat  of  mutually  exclusive  domains.  Each  domain  corresponds  to  a  particular  path 
in  tha  program.  In  tha  sense  that  tha  tat  of  input  data  points  In  that  domain  will 
cause  tha  corresponding  path  to  ba  executed.  Tha  strategy  proposed  Is  path  oriented) 

In  tatting  a  particular  path,  we  are  actually  tasting  the  computations  performed  by 
tha  program  over  a  specific  Input  spaca  domain. 

Given  a  particular  path,  tha  fora  of  tha  boundary  of  tha  corresponding  domain  It  com¬ 
pletely  determined  by  the  predicates  in  the  control  statements  encountered  in  the  path. 
Thus,  an  error  in  such  a  predicate  will  be  reflected  as  a  shift  in  the  boundary  of  the 
corresponding  domain.  The  testing  strategy  to  be  described  tests  a  path  for  domain 
errors,  1  a,  detects  domain  boundary  shifts  by  observing  the  output  values  for  a  finite 
nixxber  of  teat  data  having  a  prescribed  geometrical  relationship  to  the  entire  domain 
and  Its  boundary.  These  output  valies  are  computed  by  executing  the  sequence  of  assign¬ 
ment  statements  constituting  the  path.  The  method  requires  no  Information  other  than 
the  successfully  compiled  program  for  selecting  effective  test  data.  Thus  the  problem 
has  been  converted  from  its  usual  fora  as  an  informal  study  of  programs  and  programming 
to  •  more  formal  Investigation  of  the  geometry  of  Input  apace  domains. 

Th«  strategy  Is  Initially  described  for  the  case  of  linear  predicates  ,nd  a  two-dimen¬ 
sional  Input  space.  For  the  linear  case.  It  is  shown  that,  under  appropriate  assumpt¬ 
ions,  the  number  of  test  points  to  reliably  teat  a  domain  grows  only  linearly  with  the 
n\mbmr  of  predicates  along  the  path  and  with  the  dimensionality.  The  techniques  are 
then  extended  to  K  dimensions,  and  various  other  extensions  are  considered,  Including 
non-linear  predicates. 

A  domain  boundary  error  analysis  is  presented,  which  Is  helpful  In  choosing  the  best 
loc*tlons  for  test  points.  The  application  of  the  domain  strategy  in  discrete  spaces 
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1*  analysed  to  study  ths  effect  of  roundoff  srror  In  selecting  tast  points. 

In  tha  concluding  section  a  n unbar  of  opan  quastions  generated  by  this  Investigation 
ara  prasanted.  and  tha  prospacts  for  tha  practical  application  of  tha  domain  testing 
strategy  ara  avaluatad. 


In  ordar  to  investigate  domain  arrors,  we  naad  to  consldar  tha  language  In  which  pro¬ 
grams  will  ba  wrlttan.  Tha  control  structuras  should  ba  simple  and  conclaa,  and  should 
rasaadils  thoss  avallabla  in  most  procsdura  orlantad  languages.  Tor  simplicity  wa 
assuma  a  single  real-valued  data  typa.  and  this  is  convartad  to  Integer  valuas  for  usa 
as  00-loop  indicas.  Bacauaa  this  is  a  path  orlantad  approach,  no  astra  control  flow 
pr ob lams  ara  lntroducad  by  block  structure.  Thus  no  provision  la  aada  for  block  struct¬ 
ure.  as  it  would  only  add  astra  book-keeping  to  keep  track  of  local  variables  and  block 
invocation  or  aslt. 

A  number  of  programming  language  features  ara  assumed  not  to  occur  In  tha  programs  wa 
ara  to  analyse  for  domain  errors.  The  first  feature  is  that  of  arrays;  despite  tha 
fact  that  arraya  coaoonly  occur  m  programs,  a  predicate  which  refers  to  an  element  of 
an  Input  array  can  causa  major  complications  (Ramaooorthy  (011)1.  A  second  class  of 
language  features  which  will  ba  eacluded  in  our  analysis  Is  that  of  subroutines  and 
functions.  Tha  problems  of  side  effects  and  of  parameter  passing  pose  difficulties 
for  domain  testing.  The  third  class  of  features  which  are  not  currently  analysed  by 
domain  testing  include  non-nuaerica 1  data  types  such  as  character  data  and  pointers. 
These  are  admittedly  very  Important  features,  and  further  research  is  needed  to  invest¬ 
igate  whether  these  features  pose  any  fundamental  limitations  to  the  domain  tasting 
strategy. 

Since  Input/output  (I/O)  processing  la  so  closely  linked  to  a  machine  or  ccxspller  en¬ 
vironment,  we  will  assure  that  all  I/O  errors  have  previously  been  eliminated.  Thus 
only  tha  moat  elementary  I/O  capabilities  ara  provided;  input  Is  provided  by  a  slmpla 
Rl*0  statement,  and  output  is  accomplished  with  a  slmpla  WRITE  statement. 

The  types  of  control  flow  constructs  Investigated  in  this  research  Include  sequence , 
alternation,  and  Iteration  control.  Slnca  tha  analysis  Is  path  orlantad,  GOTO  state¬ 
ments  could  be  included  without  adversely  affecting  any  results,  except  that  program 
paths  could  become  quite  complex. 

All  computation  la  accomplished  by  means  of  arithmetic  assignment  statements  which  also 
provide  tha  basic  sequential  flow  of  control.  In  each  etatement  a  single  variable  la 
assigned  a  value.  Tha  nqht  hand  side  of  an  assignment  atataaeent  la  an  arithmetic 
axprasalon  using  variables,  constants,  and  a  sat  of  basic  arithamtlc  operators 
(♦.  *,  /). 

Tha  general  pradlcata  form  uaed  for  control  flow  la  a  Boolean  combination  of  arithmetic 
relational  expressions.  Tha  logical  operators  OR  and  ARO  are  used  to  fora  thasa  Boolean 
combinations .  each  arithmetic  relational  expression  contains  s  relational  operator 
from  tha  sat  (<,  >,  • .  «,  >,  p) .  These  operators  form  s  complata  sat,  and  thus  the 
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logical  operator  NOT  is  unnecessary.  If  a  predicate  consists  of  two  or  more  relational 
expressions  with  Booiaan  oparatora,  than  it  la  a  compound  prodicsts.  A  simplm  prodie- 
sts  conalata  of  )uat  a  single  ralatlonal  expression. 

Tha  alternation  type  of  control  flow  la  achieved  by  using  the  IF-THEN-ELSE-ENOIF  con- 
atruct.  The  conditional  aaaociated  with  the  IF  statement  la  a  general  predicata.  Any 
well-formed  program  segment,  including  the  null  program  segment,  can  be  used  In  tha 
THEN  and  ELSE  portlona  of  the  IF  construct.  The  EN01F  statement  la  just  a  delimiter 
for  the  IF  construct,  which  clarlfiaa  tha  nestinq  structure  and  eliminates  any  potent¬ 
ially  ambiguous  ELSE  clausa. 

A  general  Iteration  conatruct  la  Included  which  consists  of  a  00  statement,  loop  body, 
and  EN000  delimiter.  The  00  statement  can  be  in  one  of  three  format 

e  00  I  •  IN  I T ,  final,  INCt; 
e  00  UHIlE  (general  predicate)! 

e  00  I  •  INIT,  FINAL,  INCN  WHILE  (general  predicate!. 

The  loop  body  can  be  any  well-formal  program  segment,  and  the  END00  la  just  a  delimiter 
to  clarify  the  scope  of  the  Iteration. 

The  variables  used  in  a  program  are  divided  into  three  classes.  If  a  variable  appears 
in  a  NEA0  or  UNITE  statement.  It  la  classified  as  an  input  or  output  variable  respect¬ 
ively;  all  other  variables  are  called  ptoprsm  variables.  In  order  to  produce  a  clear 
delineation  between  the  three  types  of  variables,  we  assume  that  a  given  variable  be¬ 
longs  to  only  one  of  the  above  three  classes. 


Program  paths  ar.d  path  predicates 

A  program  can  be  represented  as  a  directed  graph  C  -  (V.A) ,  where  V  is  a  set  of  nodes 
and  A  Is  the  set  of  arcs  or  directed  edges  between  nodes.  In  the  language  Just  dis¬ 
cussed,  we  have  defined  a  set  of  basic  program  elements  which  consists  of  a  NEA0, 

UNITE,  assignment,  IF,  and  00  statement,  together  with  the  ENQIF  and  END00  delimiters. 
The  directed  graph  representation  of  a  program  will  contain  a  node  for  each  occurrence 
of  a  basic  program  element,  and  an  arc  for  each  possible  flow  of  control  between  these 
elements.  While  THEN  and  ELSE  statements  do  not  explicitly  appear  In  the  digraph,  the 
actions  associated  with  them  will  be  represented  as  nodes  in  the  digraph. 

A  wait  In  a  digraph  Is  defined  as  an  alternating  sequence  of  nodes  and  arcs  (V,,  Ai,i. 

V,.  A,,,, . \-l  A  VN*  •uch  that  each  arc  is  directed  from  node  Vj  to  vl4j* 

A  control  pstn  is  then  defined  to  be  a  walk  In  the  directed  graph  beginning  with  the 
node  for  the  Initial  statement  and  terminating  with  the  node  for  the  final  statement. 

It  should  be  noted  that  two  walks  which  differ  only  In  the  number  of  times  a  particular 
loop  In  the  program  Is  executed  will  be  defined  as  two  distinct  control  paths.  Thus 
the  nunber  of  control  paths  In  a  program  can  be  infinite. 

Every  branch  point  of  the  program  Is  associated  with  a  general  predicate.  This  predi¬ 
cate  evaluates  to  true  or  false,  and  Its  value  determines  which  outcome  of  the  branch 
will  be  followed.  A  predicate  la  generated  each  time  control  reaches  an  IF  or  00 
statesient  In  the  given  language.  The  path  condition  Is  the  compound  condition  which 
must  be  satisfied  by  the  Input  data  point  in  order  for  the  control  path  to  be  executed. 
It  Is  the  conjunction  of  the  Individual  predicate  conditions  which  are  generated  at 
each  branch  point  along  the  control  path.  Not  all  the  control  paths  that  exist 
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syntactically  within  tha  program  are  executable.  If  Input  data  exist  which  satisfy  tne 
path  condition,  the  control  path  Is  also  an  otcution  path  and  can  be  used  in  testing 
the  program.  If  the  path  condition  is  not  satisfied  by  any  input  value,  the  path  is 
said  to  be  infeaajMe,  and  Is  of  no  use  in  testing  the  program. 

A  simple  predicate  Is  said  to  be  immar  In  variables  V|,  V, .  Vn  If  It  Is  of  the 
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A, V,  ♦  A,V,  ♦  ....  ♦  AnVn  HOP  K , 

where  K  and  tha  A^  are  constants,  and  HOP  represents  one  of  the  relational  operators 
(<,>,«, «. ». p) .  A  compound  predicate  Is  linear  when  each  of  its  component  simple  predi¬ 
cates  is  linear. 

In  general,  predicates  can  be  expressed  In  terms  of  both  program  variables  and  Input 
variables.  However,  in  generating  Input  data  to  satisfy  tha  path  condition  we  must  wor*- 
wlth  constraints  only  In  terms  of  input  variables.  If  we  replace  each  program  variable 
appearing  in  the  predicate  by  its  symbolic  value  in  terms  of  input  variables,  we  get 
an  egulvalent  constraint  which  we  call  the  predicate  i n t er prec a t i on .  A  particular 
Interpretation  is  egulvslent  to  the  original  predicate  m  that  input  variable  values 
satisfying  the  interpretation  will  lead  to  tho  computation  of  program  variables  which 
also  satisfy  the  original  predicate.  A  tingle  predicate  can  have  many  different  inter¬ 
pretations  depending  upon  which  path  is  selected,  for  each  path  will  in  general  consist 
of  a  different  sequence  of  assignment  statements.  The  following  program  segment  pro¬ 
vides  example  predicates  and  interpretations. 

*£  AD  A, Si 

IF  A  >  B 

TH£H  C  •  B  *  ll 
Cist  C  •  B  -  ll 
IBOIfj 

D  -  J*A  ♦  B| 

IF  C  «  0 
THIS  C  •  Oi 
CISC 

DO  I  •  l,Bi 

K  -  I  ♦  2M| 

CBOOO) 

CBOlf . 

IF  0  •  2 

THtH  f  e  (  •  At 

use  r  •  i  -  Ai 

CHOI F I 
MBIT!  Pi 

In  the  first  predicate,  A  >  B,  both  A  and  B  are  AnPu*  variables,  so  there  Is  only  one 
Interpretation.  The  second  predicate,  C  <  0,  will  have  two  interpretations  depending 
on  which  brsnch  was  taken  in  the  first  IF  construct.  Tor  paths  on  which  the  INCH 
C  ■  B  ♦  1  clause  Is  executed  the  interpretation  Is  B  «  1  s  0  or  equivalently  B  s  -1. 

Mhen  the  CISC  C  •  B  -  1  branch  is  taken,  the  interpretation  la  B  -  1  «  0,  or  equivalently 
B  4  I.  within  the  second  I F  -  INCH -C ISC  clause,  a  nested  00- loop  appears.  The  00-loop 
Is  executed) 
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No  times  If  B  <  l 
ones  If  1  <  B  <  2 
twice  If  2  «  B  <  3 
•  tc. 

Thu*  the  selection  of  a  path  will  require  *  specif icatlon  of  the  number  of  tines  that 
the  00- loop  Is  executed,  and  a  corresponding  predicate  is  applied  which  selects  those 
input  points  which  will  follow  that  particular  path.  Even  though  the  third  predicate, 
D  *  2.  appears  on  four  different  paths,  it  only  has  one  Interpretation,  2*A  ♦  B  •  2, 
since  D  is  assigned  the  v  lue  2*A  -Bln  the  same  statement  In  each  of  the  four  paths. 


irtance  of  linear  predicates 


The  domain  testing  strategy  becomes  particularly  attractive  from  a  practical  point  of 
view  If  the  predicates  are  assumed  to  be  linear  In  input  variables.  It  might  teem  to 
be  an  undue  limitation  to  require  that  predicate  interpretations  be  linear  for  the  pro¬ 
posed  strategy.  In  fact,  however,  as  the  following  discussion  shows,  this  represents 
no  real  limitation  for  many  important  applications. 


A  number  of  authors  have  provided  data  to  show  that  simple  programming  language  con¬ 
structs  are  used  store  often  than  complex  constructs.  Knuth  (010)  studied  a  random 
sample  of  rORTAAN  programs  and  found  that  861  of  all  assignment  statements  were  of  the 
forms  i 


Vi  •  V,. 

V,  •  V,  ♦  V,, 

or  v,  -  v,  -  v,. 

Also  701  of  all  00  loops  In  the  programs  contained  less  than  four  statements.  Elshoff 
'005 .DOS)  atudled  120  production  PL/1  programs  and  showed  similar  results,  including 
the  fact  that  »7»  of  all  arithmetic  operator*  are  •  or  -.  and  981  of  all  expressions 
contain  fewer  than  two  operator*. 

An  experiment  of  particular  relevance  to  the  present  context  is  reported  in  Cohen  (004) 
using  typical  data  processing  programs,  since  program  functions  and  programming  practice 
tend  to  be  reasonably  uniform  in  this  area.  A  random  sample  of  50  COBOL  programs  was 
taken  directly  from  production  data  processing  application*  for  this  study.  In  this 
ststlc  analysis  each  predicate  is  classified  according  to  whether  it  1*  linear  or  non¬ 
linear,  and  the  number  of  input  variables  used  in  the  predicate  has  also  been  recorded. 
In  addition,  the  number  of  input-independent  predicates  were  tabulated,  since  these 
predicate*  do  not  produce  any  input  constraints.  The  number  of  equality  predicates  Is 
also  reported  since  these  predicates  are  very  beneficial  in  reducing  the  number  of  test 
points  required  for  a  domain.  These  data  are  summarised  in  Figure  1. 

The  most  important  result  is  that  only  on*  predicate  out  of  the  1225  tabulated  in  the 
study  can  possibly  be  a  non-linear  predicate.  The  predicates  are  also  very  simple  since 
most  of  them  refer  to  only  on*  input  variable,  and  no  predicate  in  this  sample  uses 
more  than  two  input  variables. 

In  conclusion,  while  this  study  by  no  mean*  represents  an  exhaustive  survey,  v*  believe 
the  sample  is  large  enough  to  indicate  that  non-linear  predicate  interpret ations  are 
rarely  encountered  m  data  processing  spplications.  It  is  clear  that  any  testing 
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Total  predicates 

1228 
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Linear  predicates 
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0-104 

Non-hnear  predicates 

1 
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0-1 

Input-Independent  predicates 
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1 
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Predicates  with  1  variable 
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Predicates  with  2  variables 
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2-5 

0-20 

Equality  predicates 
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15-8 
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strategy  restricted  to  linear  predicates  la  still  viable  in  many  areas  of  programming 
practice. 


Input  space  structure 

A  program  which  has  N  input  variables  and  produces  M  output  variables  computes  a  funct¬ 
ion  which  maps  points  In  the  N-dlmens lona  1  Input  space  to  points  in  the  N-d imens lona 1 
output  space.  The  input  space  Is  partitioned  into  a  set  of  domains.  Each  domain 
corresponds  to  a  particular  executable  path  In  the  program  and  consists  of  the  Input 
data  points  which  cause  the  path  to  be  executed.  Here  formally,  an  input  apace  domain 
la  defined  as  a  set  of  input  data  points  satisfying  a  path  condition,  consisting  of  a 
conjunction  of  predicates  along  the  path.  In  this  discussion,  these  predicates  are 
assumed  to  be  simple:  compound  predicates  will  be  discussed  later. 

Me  assume  that  the  Input  space  Is  bounded  In  each  direction  by  the  minimum  and  maximum 
values  for  the  corresponding  variable.  These  min-max  constraints  do  not  appear  1  r.  the 
program  but  are  aut  cxna  t  lea  1  ly  appended  to  each  path  condition.  Since  a  single  data  type 
Is  used  for  all  variables  in  our  language,  each  variable  will  have  the  same  min-max 
constraints. 

The  boundary  of  each  domain  la  determined  by  the  predicates  In  the  path  condition  and 
consists  of  border  segeents.  where  each  segment  Is  the  section  of  the  boundary  deter¬ 
mined  by  a  single  simple  predicate  in  the  path  condition.  Each  border  segment  can  be 
open  or  closed  depending  on  the  relational  operator  In  the  predicate.  A  closed  border 
segaent  la  actually  part  of  the  domain  and  is  formed  by  predicates  with  (,  »,  or  •  oper¬ 
ators.  An  of  an  bordar  segment  forms  part  of  the  domain  boundary  but  does  not  constitute 
part  of  the  domain,  and  la  formed  by  <,  >,  and  P  predicates.  Me  shall  find  It  conven¬ 
ient  to  use  the  term  bordar  operator  to  refer  to  the  relational  operator  for  the  corres¬ 
ponding  predicate. 

Since  border  segments  In  the  input  space  are  determined  by  the  particular  predicate 
interpretations  on  the  path,  the  form  of  the  segment  may  be  different  from  that  of  the 
original  predicate.  Por  example,  with  Input  variables  A  and  B,  the  linear  predicate 
A  <  C  ♦  2  can  lead  to  a  no.i-linear  border  segment,  A  <  B*B  ♦  2,  when  C  •  B*B.  Similarly, 
a  non-linear  predicate,  C  >  A*A  ♦  B,  will  produce  a  linear  border  segment,  A  »  B,  when 
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C  •  A*A  •  A.  Since  a  predicate  can  appear  on  many  paths  and  aach  path  can  execute  a 
different  sequence  of  assignment  statements  for  tha  variables  used  in  the  predicate, 
a  single  predicate  can  have  many  different  Interpretations  and  can  form  many  discon¬ 
tinuous  border  segments  for  various  doaaalna. 

Tha  total  number  of  pradicatas  on  tha  path  Is  only  an  upper  bound  on  tha  number  of 
bordar  segawnta  in  tha  domain  boundary  slnca  certain  predicates  in  tha  path  condition 
nay  not  actually  produce  bordar  segments.  An  i nput - independent  predicate  interpretat¬ 
ion  Is  ona  which  reduces  to  a  relation  between  constants,  and  slnca  It  la  either  true 
or  falsa  regardless  of  tha  input  values,  it  does  not  further  constrain  tha  domain.  A 
redundant  predicate  ,  interpreter  ion  la  ona  which  Is  superseded  by  tha  other  predicate 
interpretations,  1  a,  the  domain  can  be  defined  by  a  strict  subset  of  the  predicate 
Interpretations  for  that  path. 

The  general  form  of  a  simple  linear  predicate  interpretation  1st 

A i  *1  >A|  I,  *  ....  •  A  X  BOP  X, 

n  n 

where  BOP  Is  the  relational  operator,  X^  are  input  variables,  and  A^ ,  K  are  constants. 
However,  the  border  segment  which  any  of  these  predicates  defines  is  a  section  of  the 
surface  defined  by  the  equalltyi 

A,  X,  ♦  A,  X,  ♦  -  ♦  A  Xn  •  X. 

n  n 

since  this  Is  the  limiting  condition  for  the  points  satisfying  the  predicate.  In  an 
N-dlmensional  space  this  linear  equality  defines  a  hyperplane  which  Is  tha  N-dlmenslonal 
generalisation  of  a  plane. 

Consider  a  path  condition  composed  of  a  conjunction  of  simple  predicates.  These  predi¬ 
cates  can  be  of  three  basic  typesi  equalities  (•),  Inequalities  <<.  >,  «,  ») ,  and 
nonequalltles  Id).  The  use  of  each  of  the  three  types  results  In  a  markedly  different 
affect  on  the  domain  boundary.  Each  equality  constrains  the  domain  to  lie  In  a  partic¬ 
ular  hyperplane,  thus  reducing  the  dimensionality  of  the  domain  by  one.  The  set  of 
Inequality  constraints  then  defines  a  region  within  the  lower  dimensional  space  defined 
by  the  equality  predicates. 

The  nonequallty  linear  constraints  define  hyperplanes  which  are  not  part  of  the  doauln, 
giving  rise  to  open  border  segments  as  mentioned  earlier.  Observe  that  the  constraint 
A  M  is  equivalent  to  the  compound  predicate  (A  <  B)  OB  (A  »  •) .  In  this  form  it  is 
clear  that  the  addition  of  a  nonequallty  predicate  to  a  set  of  Inequalities  can  spilt 
the  doeuln  defined  by  those  Inequalities  Into  two  regions. 

The  following  example  should  clarify  the  concepts  discussed  above, 

KAO  I.Ji 
C  •  I  ♦  1M  -  li 

in)  ir  c  »  * 

THIB  D  e  C  -  Xl 
CISC  D  *C  ♦  1  •  J  ♦  li 
CBOIf » 
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Plfurs  J  shows  ths  corresponding  input  spscs  partitioning  structurs  (or  this  proqraa. 


The  Input  space  i<  In  teraa  of  Inputs  I  and  J,  and  la  arbitrarily  constrained  by  the 
following  aln-max  conditional 

-1  «  I  «  4,  -I  <  >  4  *. 

Each  border  In  rigure  2  Is  labelled  with  the  corresponding  predicate,  and  each  domain 
is  labelled  with  the  corresponding  path.  The  path  notation  Is  based  upon  which  branch 
(T  or  E)  la  taken  in  each  of  the  three  IF  constructs,  a  g,  TEE. 

The  first  predicate  PI,  C  >  4,  will  be  interpreted  as  I  *  2*J  >  7  since  C  •  I  ♦  2*J  -  1. 
This  single  Interpretation  PI  Is  seen  In  Figure  2  as  a  single  continuous  border  eegsiant 
across  the  entire  Input  apace. 

The  second  predicate  P2  demonstrates  the  effects  of  both  equality  and  nonequality  predi¬ 
cates.  Domains  for  paths  through  the  The*  branch  are  constrained  by  the  equality,  and 
this  reduction  in  dimensionality  la  seen  in  the  fact  that  these  donalns  consist  of  the 
points  on  the  solid  line  segments  ETT  and  TTT .  Paths  through  the  fLSE  branch  are  con¬ 
strained  by  a  nonequallty  predicate,  and  the  corresponding  domains  consist  of  the  two 
regions  on  either  side  of  the  solid  line  segments  (e  g,  CEE).  This  predicate  has  two 
interpretations  depending  upon  the  value  assigned  to  D  and  produces  two  discontinuous 
border  segments  ETT  and  TTT. 

The  third  predicate  Pi  might  have  four  different  interpretations,  but  only  one  border 
segment  appears  In  the  diagram.  The  other  three  interpretations  do  not  produce  borders 
since  they  are  either  redundant.  Input  -  independent ,  or  correspond  to  infeasible  paths, 
with  three  IF  constructs  we  have  eight  control  paths,  but  the  diagram  contains  only 
five  domains  since  three  of  the  paths  are  infeasible,  also  many  of  these  domains  have 
fewer  than  three  border  segments  because  of  redundant  and  input  -  independent  Interpretat¬ 
ions.  From  this  example  we  cen  conclude  that  the  input  space  partitioning  structure 
of  a  program  with  many  predicates  and  a  larger  dimensional  input  space  can  be  extremely 
complicated. 

The  foregoing  definitions  and  the  example  allow  us  to  characterlte  more  precisely  domains 
which  correspond  to  simple  linear  predicate  interpretations.  For  s  formsl  ststement  of 
the  characterisation,  we  need  the  following  definitions,  A  set  Is  convex.  If  for  any 
two  points  In  ths  sat,  ths  line  segment  Joining  these  point*  la  also  In  the  set.  A 
convex  polyhedron  It  the  set  produced  by  the  Intersection  of  the  set  of  points  satisfy¬ 
ing  s  finite  nuaiber  of  linear  equslitlss  and  Inequalltlas. 

Prep. allies  t 

r or  an  execution  path  with  a  aet  of  slmpl*  linear  equality  or  inequality  predicate 
interpretations,  the  input  space  domain  Is  a  tingle  convex  polyhedron.  If  one  or  more 
simple  linear  nonequallty  predicate  Interpretations  are  added  to  this  set,  then  the  Input 
apace  domain  consists  of  the  union  of  a  set  of  disjoint  convex  polyhedre. 


The  baste  Ideas  behind  the  classification  of  errors  that  we  use  ere  due  to  Howden  (004) 


but  our  approach  to  defining  them  la  aaaowhat  aora  operational  than  that  given  in  hla 
paper. 

Proa  tha  pravloua  sactlona.  It  la  clear  that  a  program  can  be  viewed  aai 

1  Eatabl lahinq  an  eahauative  partition  of  tha  Input  apace  Into  autually  eaclualva 
doaalna  each  of  which  correaponda  to  an  eaecutable  path 

2  Specifying,  for  each  doaain,  a  aet  of  aaalgnaent  atateaenta  which  conatltute  tha 
doauln  c deputation. 

Thua  we  have  a  canonical  representation  of  a  program,  which  la  a  (poaalbly  Infinite) 
aet  of  palra  (  (0,  i  {  , )  ,  (0,  i  f , )  ,  ...  (D^ifj)...),  where  la  the  1-th  doaMln,  and  ft  la 
tha  correapondlng  doaain  coaputatlon  function. 

Clean  an  Incorrect  prograa  P,  let  ua  conalder  tha  changea  in  ita  canonical  repraaentat- 
lon  aa  a  reault  of  aodlflcatlona  perforated  on  P.  It  la  aaauatad  that  thaaa  nodi  float  lona 
are  Bade  ualng  only  peraiaalble  language  conatructa  and  reault  In  a  legal  prograa. 

Oaf  In  It  ion  I  A  domain  boundar y  modification  occura  If  the  aodlflcatlon  reaulta  in  a 
change  In  the  coeponent  of  aoae  (D^if^)  pair  In  the  canonical  repreaentatlon. 

Definition!  A  domain  computation  aodlflcatlon  occura  If  the  aodlflcatlon  reaulta  In 
a  change  In  the  tt  coeponent  of  aoae  f 0^ » f 1 )  pair  In  the  canonical  repreaentatlon. 

Definition!  A  eisaing  path  Bonification  occura  If  tha  aodlflcatlon  reaglte  In  tha 
creation  of  a  new  co^ i f 4 )  pair  auch  that  D(  la  a  aubaet  of  D,  occurring  in  aoae  pair 
(D^ifj)  In  the  canonical  repreaentatlon  of  P,  and  f^  dlffara  free  f ^ . 

Notice  that  a  particular  aodlflcatlon  (aay  a  change  of  aoae  aaalgnaent  atateaent)  can 
be  a  aodlflcatlon  of  nore  than  one  type.  In  particular,  a  alaalng  path  aodlflcatlon 
la  alao  a  domain  boundary  aodlflcatlon. 

Tha  arrora  that  occur  In  a  prograa  can  be  claaalfled  on  the  baala  of  the  modi f lcatlona 
needed  to  obtain  a  correct  prograa  and  conaeguent  changea  In  the  canonical  repreaent- 
atlon.  In  general,  there  will  be  many  correct  prograna.  and  multiple  waya  to  get  a 
particular  correct  prograa.  Hence,  the  error  claaalf lcatlon  la  not  uniqua,  but  rela¬ 
tive  to  the  particular  correct  prograa  that  would  reault  froa  the  aarlea  of  andlflcatlona. 

Definition!.  An  incorrect  prograa  P  can  be  viewed  aa  having  a  domain  error  (coapetatienal 
error)  (aieeieg  path  error)  If  a  correct  prograa  P*  can  be  created  by  a  aequenee  of 
aodlflcatlona,  at  laaat  one  of  which  la  a  doaain  boundary  aodlflcatlon  (doaain  confutat¬ 
ion  aodlflcatlon)  (alaalng  path  aodlflcatlon). 

Several  reaurka  are  In  order.  The  operational  conaequence  of  the  phraae  'can  be  viewed 
aa*  In  tha  above  definition  la  that  the  error  claaalf lcatlon  la  relative  not  only  to  a 
particular  correct  prograa,  but  alao  to  a  particular  aequenee  of  aodlflcatlona.  Por 
inatance,  conalder  an  error  In  a  predicate  Interpretation  auch  that  an  Incorrect  relat¬ 
ional  operator  la  employed ,  a  g,  uae  of  >  lnatead  of  «.  Thlo  could  be  viewed  aa  a 
doaMln  error,  leading  to  a  aodlflcatlon  of  the  predicate,  or  aa  a  confutation  error, 
leading  to  a  aodlflcatlon  of  tha  functlona  computed  on  the  two  branchee.  Tha  fact  that 
It  eight  be  aore  profitable  to  change  the  relational  operator  rathar  than  tha  function 
computation*  ia  a  conaequence  of  the  language  conatructa,  and  la  not  directly  captured 
It  the  definition*  of  tha  type*  of  error.  In  thla  paper  we  would  regard  an  error  duo 
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to  an  incorrect  relational  operator  aa  a  domain  arron  It  la  a  simpler  modification 
to  change  the  ralatlonal  operator  In  tna  pradlcata  than  to  Interchange  tha  aat  of  asalgn- 

mant  atatamanta. 

Nora  apaclflc  charactarliatlona  of  thaaa  arr ora  can  ba  made  In  tha  context  of  tha 
apaclflc  programming  language  which  wa  hava  Introduced.  In  particular,  tha  following 
Informal  daacrlption  directly  ralataa  tha  domain  and  Biasing  path  arrora  to  tha  prodl- 
cata  conatructa  allowed  In  tho  language. 

A  path  contama  a  domain  error  If  an  error  in  roam  pradlcata  Interpretation  cauaaa  a 
border  segment  to  ba  'ahlftad'  from  lta  correct  poaltlon  or  to  hava  an  Incorrect  border 
operator.  A  domain  error  can  ba  cauaed  by  an  Incorrectly  apaclflad  pradlcata  or  by  an 
incorrect  aaelgnment  atatamant  which  affacta  a  variable  uaad  In  tha  pradlcata.  An  In¬ 
correct  pradlcata  or  araignment  atatamant  may  affect  many  predicate  Interpretation* 
and  consequently  cauaa  more  than  one  border  to  ba  In  error. 

A  path  contalna  a  miaalng  path  error  when  a  pradlcata  la  alaalng  which  would  aubdlvlda 
tha  domain  and  create  a  new  execution  path  for  one  of  tha  aubdomaina.  Thla  type  of 
error  occura  whan  aome  special  condition  requiring  different  processing  la  omitted. 


fundamental  limitations 

Finite  tasting  strategies  are  fundamentally  limited  by  their  inability  to  detect  phenom¬ 
ena  occurring  in  regions  which  have  tero  volume  or  measure  relative  to  the  input  space 
or  dosuln.  Tha  first  of  these  limitations  we  ahall  define  at  coincidental  correctness. 
In  tasting  each  domain  for  tha  correctness  of  lta  boundaries.  If  the  output  for  a  tast 
case  la  correct.  It  could  ba  either  that  the  teat  point  was  In  the  correct  domain,  or 
that  It  was  In  a  wrong  domain  but  the  compulation  m  that  domain  coincidentally  yielded 
a  correct  value  for  tha  tast  point.  Similarly,  a  domain  computation  could  correspond 
to  an  incorrect  function,  but  Its  output  nay  coincide  with  tha  correct  value  for  a 
particular  test  point.  To  be  absolutely  certain  that  the  values  are  not  coincidentally 
correct.  It  would  be  necessary  to  exhaustively  test  all  the  points  of  the  domain. 

Tha  essence  of  the  coincidental  correctneaa  problem  Is  the  tame  as  that  of  the  problem 
of  deciding  If  two  arbitrary  ccsaputations  are  equivalent;  the  latter  problem  is  known 
to  be  generally  undecldable.  However,  in  practice,  the  severity  of  the  problem  is  re¬ 
lated  to  the  probability  that  for  an  arbitrary  point  this  coincidence  would  occur.  If 
the  set  of  points  for  which  the  two  functions  have  the  same  value  Is  of  measure  xero, 
then  this  probability  is  tero,  even  though  coincidental  correctness  is  still  possible. 
So.  even  with  coincidental  correctneaa  as  a  possibility,  a  testing  strategy  can  be 
almost  reliakie  in  the  sense  of  Howden  (009).  If  It  would  be  reliable  In  the  absence  of 
coincidental  correctness,  and  the  set  of  points  which  are  coincidental ly  correct  has 
xero  volume  relative  to  the  domain  being  tested. 

Another  basic  limitation  relates  to  missing  path  errors.  When  the  subdomain  associated 
with  a  aisalng  path  is  a  region  of  lower  dimensionality  than  the  original  domain,  a 
attains  path  error  of  rad teed  dimensionality  occurs.  This  typically  happens  when  the 
■lasing  predicate  la  an  equality.  If  all  that  is  available  Is  Just  the  (Incorrect) 
program  to  be  tested,  then  the  probability  that  a  finite  set  of  test  points  would  de¬ 
tect  the  aisalng  predicate  Is  xero,  since  the  volume  of  the  subdomain  Is  tero  relative 
to  that  of  the  original  domain. 

The  proposed  approach  is  capable  of  detecting  many  kinds  of  missing  path  errors,  but 
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for  iom  of  them  the  number  of  required  teat  point*  1*  Inordinate.  Hanct,  in  tho  n  i) 
section,  whar#  we  describe  the  testing  Strategy,  we  will  amply  assume  that  no  a|ai|nj 
path  errors  are  associated  with  the  path  being  tested. 


TH1  oomis  TtSTIHG  STSATtCt 

The  domain  tasting  strategy  is  designed  to  detect  domain  errors  and  will  be  effective 
in  detecting  errors  In  any  type  of  domain  border  under  certain  conditions.  Teat  points 
are  generated  for  each  border  segment  which,  if  processed  correctly,  determine  that 
both  the  relational  operator  and  the  position  of  the  border  are  correct.  An  error  in 
the  border  operator  occurs  whan  an  incorrect  relational  operator  is  used  in  the  corrrs- 
ponding  predicate,  and  an  error  in  the  position  of  the  border  occurs  when  one  or  more 
incorrect  coefficients  are  computed  for  the  particular  predicate  interpretation.  The 
strategy  is  baaed  on  a  geometrical  analysis  of  the  domain  boundary  and  takes  advantage 
of  the  fact  that  points  on  or  near  the  border  are  most  sensitive  to  domain  errors.  A 
number  of  authors  have  made  this  observation,  e  g,  Boyer  at  *1  (001)  and  Clarke  ( 00 Z ) . 

As  stated  in  Ptopotit ion  1,  a  domain  defined  by  simple  linear  predicates  is  a  convex 
polyhedron,  and  each  point  can  be  classified  according  to  its  position  within  the  do¬ 
main.  An  interior  point  is  defined  as  on*  which  Is  surrounded  by  an  c -na lghbourhood 
containing  only  points  in  the  domain.  Similarly,  a  boundtru  point  Is  one  for  which 
every  (-neighbourhood  contains  both  points  in  the  domain  and  points  lying  outside  of 
the  domain,  finally,  an  eetreee  point  is  a  boundary  point  which  does  not  lie  between 
any  two  distinct  points  in  the  domain. 

In  the  previous  section,  a  comparison  was  made  between  the  given  program  and  a  corres¬ 
ponding  correct  program;  indeed  domain  errors  were  defined  In  terms  of  this  correspond¬ 
ence.  It  should  be  emphasised  that  the  domain  strategy  does  not  require  that  the  corr¬ 
ect  program  be  given  for  the  selection  of  teat  points,  since  only  lnfonaatlon  obtained 
from  the  given  program  is  needed.  However,  It  will  be  convenient  to  be  able  to  refer 
to  a  ‘correct  border',  although  it  will  not  be  necessary  to  have  any  knowledge  about  this 
border.  Define  the  firm  »ore*r  as  that  corresponding  to  the  predicate  interpretation 
for  the  given  program  being  tested,  and  the  eottoct  sordar  as  that  border  which  would 
be  calculated  in  some  correct  program. 

The  domain  testing  strategy  la  first  developed,  explained,  and  validated  in  detail  under 
a  set  of  simplifying  assumptions) 

1  Coincidental  correctness  does  not  occur  for  any  teat  case.  If  correct  output  results 
are  produced,  w*  can  assume  that  the  teat  point  is  in  the  correct  daamln  rather  then 
being  coincidentally  correct  In  another  doamln. 

2  A  missing  path  error  Is  not  associated  with  the  path  being  tested.  Mleslng  path  errors 
of  reduced  dimensionality  pose  a  theoretical  limitation  to  the  reliability  of  any 
program  tasting  methodology. 

1  tech  border  is  produced  by  a  simple  predicate. 

4  The  path  corresponding  to  each  adjacent  domain  computes  a  different  function  than  the 
path  being  tested. 

5  The  given  border  Is  linear,  and  if  it  is  incorrect,  the  correct  border  is  also  linear. 
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4  The  input  space  la  continuous  rathar  than  discrete. 

7  Each  border  Is  produced  by  an  Inequality  predicate.  * 

•  The  Input  space  Is  two-dimensional,  corresponding  to  a  program  which  reads,  at  aoat, 
two  input  variables. 

The  first  two  assumptions  were  thoroughly  explored  In  the  previous  section,  assumptions 
)  through  •  are  for  convenience  in  the  initial  exposition,  and  we  shall  Investigate 
later  the  conditions  under  which  each  can  be  relaxed,  also,  references  (001)  and  (004) 
diacuaa  both  the  domain  strategy  and  these  assumptions  In  greater  detail. 


The  two-dimensional  linear  case 

Given  essumptlons  1  to  I,  a  set  of  test  points  Is  first  defined  for  detecting  border 
shifts,  and  then  we  shall  show  that  this  set  of  points  also  detects  all  possible  relat- 
lonal  operator  errors.  Since  the  present  analysis  is  limited  to  linear  borders  In  a 
two-dimensional  input  space,  each  border  la  a  line  segment.  Therefore,  the  correct 
border  can  be  determined  If  we  know  two  points  on  that  border. 

The  test  cases  selected  will  be  of  two  types,  defined  by  their  position  with  respect 
to  the  given  border,  an  os  test  point  lies  on  the  given  border,  while  an  off  test 
peiat  la  a  small  distance  t  from,  and  lies  on  the  open  side  of,  the  given  border. 
Therefore,  we  observe  that  when  testing  a  closed  border,  the  ON  test  points  are  In  the 
domain  being  tested,  and  each  OFF  test  point  is  In  some  adjacent  domain.  Conversely, 
when  testing  an  open  border,  each  ON  test  point  is  In  some  adjacent  domain,  while  the 
orr  test  points  are  In  the  domain  being  tested. 

rigure  )  shows  the  selection  of  three  test  points,  A.  B.  and  C  for  a  closed  Inequality 
border  segmnt.  In  this  and  subsequent  figures  the  small  arrows  are  used  to  indicate  the 
domain  which  contains  the  border  segment.  The  three  points  must  be  selected  In  an  ON- 
OFF  -ON  sequence.  Specifically,  if  teat  point  C  la  projected  dom  on  line  AB,  then  the 
projected  point  must  He  strictly  between  A  and  B  on  this  line  segment.  Also  point  C 
is  selected  a  distance  i  from  the  given  border  segment,  and  will  be  chosen  so  that  It 
satisfies  all  the  inequalities  defining  domain  0  except  for  the  Inequality  being  tested. 

Zt  must  be  shown  that  test  points  selected  in  thla  way  will  reliably  detect  domain  errors 
du>  to  boundary  shifts.  If  any  of  the  test  points  lead  to  an  Incorrect  output,  then 
clearly  there  is  an  error.  On  the  other  hand,  If  the  outputs  of  all  these  points  are 
correct,  then  either  the  given  border  Is  correct  or  we  have  gained  considerable  inform¬ 
ation  aa  to  the  location  of  a  correct  border.  Figure  ]  shows  that  the  correct  border 

must  lie  on  or  above  points  A  and  B,  and  must  lie  below  point  C,  for  by  assumptions  1 

and  4  (page  139),  each  of  these  test  points  must  lie  in  its  assumed  domain.  So  if  the 
given  border  Is  Incorrect,  the  correct  border  can  only  belong  to  a  class  of  line  seg- 

ments  which  Intersect  both  closed  line  segments  AC  and  BC. 

Figure  J  indicates  a  specific  correct  border  from  this  clsss  which  Intersects  line  seg¬ 
ments  AC  and  BC  at  F  and  Q  respectively.  Define  the  doeain  error  eafnitede  for  this 
correct  border  to  be  the  maximum  of  the  distances  from  F  and  from  Q  to  the  given  border. 
Then  it  la  clear  that  the  chosen  test  points  have  detected  domain  errors  due  to  border 
shifts  except  for  a  class  of  domain  errors  of  magnitude  less  than  e.  In  a  contlnuoua 
•pace  c  can  be  chosen  arbltrsnly  small,  and  as  r  approaches  xero,  the  line  segments 
AC  and  BC  becosM  arbitrarily  close  to  the  given  border,  and  In  the  limit,  we  can  conclude 
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khet  khe  given  border  la  identical  to  the  correct  border.  However,  the  continuity  of 
the  apace  alao  implies  that  reqardless  of  h«n#  aauill  c  la  choaen,  border  ahlfta  of  smg* 
nltude  leaa  than  t  may  not  be  detected,  and  therefore  we  must  corresponding ly  qualify 
our  reaulta. 

riqure  4  ahowa  the  three  general  typea  of  border  ahlfta,  and  will  allow  ua  to  aee  how 
the  OH-orr-OH  sequence  of  teat  polnta  works  In  each  caae.  In  rigura  4(a),  the  border 
ahtft  haa  effectively  reduced  domain  D,.  Teat  polnta  A  and  B  yield  correct  outputa, 
for  they  remain  in  the  correct  domain  D(  deaplte  the  ahlfted  border.  However,  the 
border  haa  ahlfted  paat  teat  point  c,  causing  It  to  be  In  domain  0.  lnatead  of  domain 
0t.  Since  the  proqram  will  now  follow  the  wrong  path  when  executing  Input  C,  Incorrect 
reaulta  will  be  produced.  In  riqure  4(b),  the  domain  D.  haa  bean  enlarqed  due  to  the 
border  ahlft.  Here  teat  point  C  win  be  proceaeed  correctly  alnce  It  la  atlll  In 
domain  0t.  but  both  A  and  B  will  detect  the  ahlft  alnce  they  ahould  alao  be  In  doaaaln 
Finally  in  riqure  4(c),  only  teat  point  B  will  be  Incorrect  alnce  the  border  ahlft 
cau***  to  l*  Oi  lnatead  of  0|.  Therefore,  the  OH-orr-OH  sequence  la  effective 
alnce  at  leaat  one  of  the  three  polnta  auat  be  in  the  wronq  domain  aa  long  aa  the  border 
ahtft  la  of  a  magnitude  greater  than  t. 

Mecall  in  riqure  I  that  we  required  the  OPP  point  C  to  aatlafy  all  the  lnequalltiea 
defining  domain  D  except  for  the  Inequality  being  tea  ted .  The  reaaon  for  thia  require* 
nent  la  that  aame  correct  border  eeqment  may  terminate  on  the  extenalon  of  an  adjacent 
border,  rather  than  lnteraectlng  both  line  aegmente  AC  and  BC  aa  we  have  argued.  Since 
we  have  aaaumed  a  continuous  apace,  C  could  alwaya  be  choaen  cloaer  to  tha  given  border 
in  order  to  aatlafy  the  adjacent  border  Inequalities. 

me  must  also  demonstrate  the  reliability  of  the  method  for  domain  errora  in  which  the 


predicate  operator  la  Incorrect.  If  the  direction  of  the  Inequality  la  wronq,  e  q, 

(  la  uaed  lnatead  of  a,  the  domalna  on  either  aide  of  the  border  are  interchanqed.  and 
any  point  In  either  domain  will  detect  the  error.  A  more  aubtle  error  occur*  when  )uat 
the  border  ltaelf  la  In  the  wronq  domain,  e  q,  (la  uaed  lnatead  of  <.  In  thla  caae 
the  only  polnta  affected  lie  on  the  border,  and  atnee  wa  wlwaya  teat  on  polnta,  thla 
type  of  error  will  alwaya  be  detected.  If  the  correct  predicate  la  an  equality,  the 
OFT  point  will  detect  the  error. 

The  doamln  teatlnq  atrateqy  require*  at  Boat  )*P  teat  polnta  for  a  domain,  where  P, 
the  nuaiber  of  border  aeqnenta  on  thla  boundary,  la  bounded  by  the  number  of  predlcatee 
encountered  on  the  path.  However,  wa  can  reduce  thla  coat  by  aharlnq  teat  polnta  be¬ 
tween  adjacent  border*  of  the  domain.  The  requirement  for  aharlnq  an  ON  point  la 
that  It  la  an  extreme  point  for  two  adjacent  bordera  which  are  both  cloaed  or  both  open 
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In  the  iiwpli  in  Figure  J,  tha  points  that  can  ba  sharad  ara  k,  .  and  *•. 
masber  of  ON  points  naadad  to  tast  tha  antlra  domain  boundary  can  ba  raducad  by  aa  such 
as  ona  halt.  1  a,  tha  nvaabar  of  tast  points,  TP,  rsqulrad  to  tast  tha  complata  domain 
boundary  lias  in  tha  following  ranget 

1*P  <  TP  <  J*P. 

Ivan  mors  significant  savings  ara  poaslbla  by  sharing  tha  tast  points  for  a  common 
bordar  between  two  adjacent  domains.  tf  both  domains  sra  taatad  lndapandant ly ,  tha 
common  bordar  batwaan  tham  Is  tastad  twica,  using  a  total  of  six  tast  points.  Xf  this 
bordar  has  shiftad.  both  domains  must  ba  affactad,  and  tha  arror  will  ba  datactad  by 
tasting  althar  domain.  Tharafora,  tha  sacond  sat  of  tast  points  can  safaly  ba  omlttad. 
However,  tha  coat  sawings  In  such  sharing  should  ba  balancad  against  tha  additional 
procasalng  ragulrad. 

Ma  now  formally  summarise  tha  results  of  this  section  In  tha  following  proposition. 

Prepesl ties  1 

Given  assumptions  1  through  •  (page  }Jf),  with  tha  OPP  tast  point  chosen  a  distance  c 
from  tha  corresponding  bordar,  tha  doauln  tasting  strategy  Is  guaranteed  to  detect  all 
domain  errors  of  magnitude  greater  than  r.  Moreover ,  tha  cost  Is  no  mors  than  1*P  tast 
points  par  domain,  where  P  Is  tha  number  of  predicates  along  tha  corresponding  path. 
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The  domain  tasting  strategy  davalopad  for  tha  two-dimensional  casa  can  ba  extended  to 
tha  general  N-dlmanalonal  casa  In  a  straightforward  mannar.  Tha  central  preparty  usad 
In  tha  pravloua  analysis  was  tha  fact  that  a  llna  is  unlqualy  datarmlnad  by  two  points. 

Wa  can  aaslly  ganarallxa  this  proparty  sinca  an  N-dimenslonal  hyparplana  Is  datarmlnad 
by  N  llnaarly  indapandant  points.  So,  wharaas  In  tha  two-dimensional  casa  wa  had  to 
ldantlfy  only  two  points  on  tha  corract  bordar,  in  qeneral  wa  hava  to  Idantlfy  N  points 
on  tha  corract  bordar,  and  In  addition,  thasa  points  must  ba  guaranteed  to  ba  llnaarly 
indapandant. 

Tha  validation  of  domain  tasting  for  tha  ganaral  llnaar  casa  Is  basad  on  tha  sasta  geo- 
matrlc  arqumants  usad  in  tha  two-dimensional  casa.  Tha  toy  to  tha  mathodology  Is  that 
tha  corract  bordar  must  lntarsact  avary  OFT -ON  llna  sagmant,  assuming  that  tha  tast 
points  ara  all  corract.  Sinew  wa  must  ldantlfy  a  total  of  N  points  on  tha  corract 
bordar,  wa  naad  N  OFF-ON  llna  sagmants,  and  wa  can  achlava  this  by  tasting  N  llnaarly 
indapandant  ON  tast  points  on  tha  givan  bordar  and  a  slngla  OFF  tast  point  whosa  pro¬ 
motion  on  tha  givan  bordar  is  a  convax  combination  of  thasa  N  points.  In  addition, 
ha  in  tha  two-dimansional  casa,  tha  OFF  point  must  also  satisfy  tha  lnaquallty  constraints 
corraspondlng  to  all  adjacent  bordars. 

Cvan  though  wa  do  not  know  thasa  spaclflc  points  at  which  tha  corract  bordar  lntarsacts 
tha  ON-OFF  sagmants,  wa  do  know  that  thasa  points  must  ba  llnaarly  Indapandant  sinca 
tha  ON  points  ara  llnaarly  Indapandant.  Tha  OFT  point  is  a  distance  c  from  tha  givan 
bordar,  and  in  tha  limit  as  t  approaches  taro,  each  orF-ON  llna  segment  becomes  arbit¬ 
rarily  close  to  tha  given  bordar.  However,  as  in  tha  two-dimensional  casa,  tha  c-llal- 
tstion  means  that  only  bordar  shifts  of  magnitude  greater  than  c  will  ba  detected. 

Tha  domain  tasting  strategy  requires  at  most  (N*1)*P  tast  points  par  domain,  where  N 
is  tha  dlmensi onall ty  of  tha  Input  space  in  which  tha  domain  is  defined  and  P  Is  tha 
number  of  bordar  segments  in  tha  boundary  of  tha  spaclflc  domain.  However,  wa  again 
can  reduce  this  tasting  cost  by  using  extrema  points  as  ON  tast  points.  Each  extreme 
point  Is  formed  by  tha  Intersection  of  at  least  N  bordar  segments,  and  therefore  one 
point  can  ba  usad  to  test  up  to  N  borders.  In  addition,  extrema  points  ara  also  lin¬ 
early  Indapandant.  Each  bordar  mutt  ba  tatted  by  N  ON  points,  and  any  points  beyond 
this  ara  redundant,  and  so  not  all  extreme  points  on  each  bordar  ara  required,  ha  a 
result  of  this  kind  of  sharing,  tha  number  of  test  points  can  ba  as  few  as  }*P.  As 
In  tha  two-dlmanslonal  casa,  there  can  be  further  savings  if  test  points  ara  shared 
between  adjacent  doaulnt.  rinally,  since  soma  of  the  P  border  segments  may  ba  produced 
by  tha  sln-sss  constraints  which  daflna  tha  bounds  of  tha  Input  space,  tha  number  of 
tast  points  can  ba  reduced  still  further.  If  wa  can  assume  that  thasa  constraints  are 
predetermined  and  naad  not  ba  tastad. 

This  generalisation  to  N  dimensions  Is  significant  since  very  few  non-trlvial  programs 
hava  only  two  input  variables,  wa  summarise  tha  results  so  far  in  tha  following  propo¬ 
sition. 

Prapas I  Stas  3 

Given  assumptions  1  to  7  (page  339),  with  tha  OrF  tast  point  chosen  a  distance  c  from 
tha  corraspondlng  bordar,  tha  domain  tasting  strategy  is  guaranteed  to  detect  all 
domain  arrora  of  magnitude  greater  than  c  regardless  of  tha  dimensionality  of  tha  In¬ 
put  space.  Moreover,  tha  cost  Is  not  more  than  (N»l> ‘Potest  points  par  domain. 


jaa 


Equality  and  nontqmluv  predicates 


Equality  predicates  constrain  the  domain  to  lla  In  a  lower  dimensional  space.  If  we 
have  an  N-dimenaional  input  space  and  the  domain  Is  constrslned  by  L  independent  equal¬ 
ities,  the  remainin')  inequality  and  nonequallty  predicates  then  define  the  domain 
within  the  (N-L) -dimensional  subspace  defined  by  the  set  of  equality  predicates. 

In  Flqure  6  we  see  the  equality  border  and  the  proposed  set  of  test  points.  In  a  gen¬ 
eral  N-dlmenslonal  domain,  let  us  first  consider  a  total  of  N  ON  points  on  the  border 
and  two  OrF  points,  one  on  either  side  of  the  border.  As  before,  the  ON  points  must 
be  Independent,  and  the  projection  of  each  OPP  point  on  the  border  must  be  a  convex 
cca^lnatlon  of  the  ON  points. 


Given  an  Incorrect  equality  predicate, 
the  error  could  be  either  In  the  re¬ 
lational  operator  or  In  the  position 
of  the  border  or  both.  The  proposed 
set  of  test  points  can  be  shown  to  de¬ 
tect  an  operator  error  or  a  position 
error  by  arguments  analogous  to  those 
previously  given.  This  set  of  points 
Is  also  adequate  for  almost  all  com¬ 
binations  of  operator  and  position 
errors,  except  for  the  following  patho¬ 
logical  possibility.  Let  us  assume  that 
the  border  has  shifted  and  tha  correct 
predicate  is  a  nonequallty.  If  both 
OPP  points  happen  to  lie  on  the  correct  border  while  none  of  the  ON  points  belong  to 
this  border,  the  error  would  go  undetected.  This  singular  situation  is  diagrammed  as 
the  dashed  border  In  Figure  7,  where  Ai  and  A,  are  the  ON  points,  and  Ci  and  C>  are 
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the  orr  points.  This  problem  can  ba  solved  by  testing  on a  additional  point  salactad 
so  that  it  lias  both  on  tha  given  bordar  and  tha  corract  border  for  this  c aaa,  1  a,  at 
tha  intarsaction  point  of  tha  given  bordar  with  tha  lina  segment  connecting  tha  two 
OFF  points.  This  additional  point  is  danotad  by  B  in  tha  figure. 

Each  equality  pradicata  can  thus  ba  completely  tested  using  a  total  of  (NO)  teat  points. 
By  sharing  test  points  between  all  tha  equality  predicates,  this  number  can  ba  consid¬ 
erably  reduced,  but  tha  reduction  depends  upon  values  of  N  and  L.  In  addition,  since 
tasting  tha  equality  predicates  reduces  tha  affective  dimensionality  to  (N-L)  for  each 
of  tha  inequality  and  nonequality  borders,  and  tha  equality  ON  test  points  can  be 
shared,  even  further  reductions  are  possible. 

for  the  case  of  a  nonequality  border,  the  testing  strategy  is  identical  to  that  of  the 
equality  border  Just  discussed.  Tha  arguments  for  the  validity  of  the  strategy  are 
analogous  to  those  in  previous  cases.  Again  in  this  case,  the  pathological  possibility 
discussed  In  connection  with  the  equality  predicate  can  occur,  and  can  be  handled  In 
the  sane  way.  The  major  difference  is  that  while  test  points  can  be  extensively  shared 
between  equality  and  inequality  borders,  in  general  such  sharing  is  not  possible  between 
nonequality  and  inequality  borders.  The  following  proposition  summarises  the  situation 
for  testing  linear  borders  in  N-dl»ensions . 

Sresoslllea  4 

C\ven  assumptions  1  through  i  (page  ]  3  9  >  .  with  each  orr  point  chosen  a  distance  c  frost 
the  corresponding  border,  the  domain  testing  strategy  is  guaranteed  to  detect  all  domain 
errors  of  siagnitude  greater  than  c  using  no  more  than  P*(N*1)  test  points  per  domain. 


An  example  of  error  detection  using  the  dentin  strategy 

The  domain  testing  strategy  has  been  described  and  validated  using  somewhat  complicated 
algebraic  and  geometric  arguments.  In  this  section  we  hope  to  complement  those  discuss¬ 
ions  by  demonstrating  how  s  set  of  doaialn  test  points  tor  a  short  sample  program  actually 
detects  specific  examples  of  different  types  of  programming  errors.  In  discussing  each 
error  we  will  focus  on  a  specific  domain  affected  by  the  error,  and  a  careful  analysis 
of  Its  effect  on  the  domain  will  allow  us  to  Identify  those  domain  test  points  which 
detect  the  error. 

The  short  example  program  reads  two  values,  I  and  J,  and  produces  a  single  output  value 
M.  Therefore,  the  input  space  is  'wo-dimens Iona  1 .  and  the  following  min-max  constraints 
have  been  chosen  so  that  the  Input  space  diagram  would  not  be  too  large  or  complicated. 

-  a  «  i  <  •  -  5  •  j  1  s. 

In  addition,  since  this  Is  a  two-dimensional  space,  we  will  also  test  extreme  points  for 
the  border  segments  produced  by  the  min-max  constraints  in  order  to  be  able  to  detect 
as  many  missing  path  errors  as  possible. 

Even  though  the  Input  space  is  assumed  to  be  continuous,  the  co-ordinates  of  each  teet 
point  are  specified  to  an  accuracy  of  0-2  In  order  to  simplify  the  diagrams  and  discuss¬ 
ions.  Of  course,  in  an  actual  implementation  each  OFF  point  would  be  chosen  much  closer 
to  the  be  der. 


he  sample  program  is  listed  below,  and  it  consists  of  three  simple  If  constructs,  the 
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flfin  h  Inpet  apace  amain  lul  point! 

first  two  of  which  srs  Inequalities  and  the  last  of  which  Is  an  equality.  Tha  Input 
apace  structure  Is  dlairammed  in  flours  I,  whara  tha  solid  diagonal  border  across  tha 
entire  space  Is  produced  by  tha  first  predicate,  tha  dashed  horlrontal  border  and  short 
vertical  border  at  I «0  are  produced  by  tha  second  predicate,  and  the  vertical  equality 
border  at  I  - 1  corresponds  to  the  third  predicate.  In  addition,  doaaaln  test  points 
have  been  indicated  for  the  two  doaaains  TTI  and  CTT  which  we  will  discuss. 

Stateeenr 
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1  If  I  «  J  •  1 
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1  (ISf  R  •  3*1  ♦  It 

1*0 1 f  i 

4  if  k  *  r  ♦  l 

s  thu  l  •  i  *  ii 

•  ELSE  L  -  J  -  li 

moifi 

7  If  X  -  J 

•  TN|H  N  •  3*L  e  Rt 

•  ELSE  N  e  L  a  3*R  -  1| 
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Figure  t  illustrates  two  '  y  pe  a  of  trrori  we  would  Ilk*  to  consider.  The  first  la  an 
•  rror  In  the  inequality  predicate  in  statement  *4  of  the  above  program.  (R  »  1*1), 
where  It  la  assumed  that  the  correct  predicate  ahould  be  (It  »  1*2).  Th  1  a  correaponda 
to  an  inequality  border  ahlft.  and  the  modified  domain  itructure  la  ah own  in  Figure  10. 
Three  polnta  have  been  aelected  to  teat  thla  border,  and  It  can  be  aeen  in  rigure  * 
that  the  two  ON  pointa  detect  thla  error,  where  M  and  M '  repreaent  the  output  varlablea 
for  the  given  proqran  and  for  the  aaaumed  correct  program  reapectively .  Note  that  aa 
a  reault  of  thla  error,  the  vertical  border  at  I *0  in  Figure  (  haa  alao  ahlfted  to  I»1 
In  Figure  10.  and  if  teated,  would  alao  reveal  thla  error. 


Figure  »  alao  ahowa  the  effect  of  an  error  In  an  equality  predicate  In  atatement  »7  of 
the  given  program.  It  la  aaaumed  that  the  correct  predicate  ahould  be  C I — 5  — J >  rather 
than  the  II»S)  predicate  which  occura  In  the  given  program.  Figure  11  ahowa  the  modi¬ 
fied  Input  apace  atructure,  and  It  can  be  aeen  that  equality  bordera  TTT  and  ETT  have 
ahlfted.  rigure  »  ahowa  the  five  pointa  which  teat  the  ETT  boarder,  and  note  that  two 
ON  pointa  both  detect  thla  ahlft. 


Figure  11  indicate*  that  the  domain  atrategy  can  alao  detect  a  computation  error  and  a 
aiaalng  path  error,  even  though  we  have  previoualy  noted  that  reliability  cannot  be 
proven  for  theae  caaea.  The  computation  error  arlaea  from  atatement  *i  in  the  given 
program,  where  it  la  aaaumed  that  the  correct  aaalgnment  atatement  for  thla  CISC  clauae 
la  ( !»•  I  - 1 )  inatead  of  (LwJ-ll  which  actually  appeara  In  the  given  program.  Since  L  la 
not  uaed  In  any  aubaequent  predicate,  thla  correaponda  to  a  computation  error  rather 
than  a  domain  error.  Thua  the  input  apace  structure  in  Figure  S  la  applicable  for  both 
the  given  and  the  correct  programs.  Figure  12  shows  the  six  test  pointa  which  have 
been  chosen  to  teat  domain  TEE  which  la  affected  by  thla  computation  error.  Four  of 
the  points  ahould  indicate  the  error,  but  note  the  teat  results  at  (-«,  -S)  are 


coincident 4 lly  correct)  the  remaining  three  points  detect  the  error. 


Suppose  In  program  statement  *2  the  THEN  clause  la  replaced  by  the  following  code. 

THIS  IF  2*J  <  -SM  -  40 
THIN  I  •  Ji 
CISC  X  »  I  ♦  3  -  1) 

EN01F) 

This  corresponds  to  s  missing  path  error  and  la  Indicated  as  such  In  Figure  12.  Fig¬ 
ure  13  shows  how  the  domain  TEE  Is  modified  by  this  missing  path  error,  but  note  that 
only  test  point  (-8,-5)  detects  this  error.  If  the  <  Inequality  In  the  siisslnq  predi¬ 
cate  had  been  an  equality,  this  would  have  produced  a  missing  path  error  of  reduced 
dimensionality,  corresponding  to  a  domain  consisting  of  Just  the  line  segment  In  Fig¬ 
ure  13,  and  would  have  gone  undetected. 


EXTENSIONS  OF  THE  DOMAIN  TESTING  STRATEGY 

Many  sasoaiptiont  were  required  in  presenting  the  previous  results,  but  to  some  extent 
these  assumptions  were  made  to  allow  a  simple  exposition  of  the  domain  testing  strategy. 
This  section  will  discuss  assumptions  3,  4,  and  J  (page  3391  which  deal  with  compound 
predicates,  adjacent  domains  which  compute  the  same  function,  and  non- 1  inear  borders, 
respectively.  The  treatment  of  these  cases  will  certainly  require  additional  test 
points,  and  in  soam  instances  will  demand  extra  processing  which  may  render  this  test¬ 
ing  approach  impractical.  However,  one  of  the  main  objectives  of  this  section  t  to 
illustrate  that  none  of  the  assumptions  3.  4,  or  5  poae  a  theoretical  limitation  to 
the  domain  testing  strategy  which  cannot  be  dealt  with  In  some  fashion. 


The  general  non-linear  case 

A  finite  domain  testing  strategy  cannot  be  effective  for  the  universal  class  of  non¬ 
linear  borders,  but  we  must  determine  whether  this  is  caused  by  some  fundamental  diff¬ 
erence  between  linear  and  non-linear  functions.  If  the  problem  is  that  we  are  consider¬ 
ing  too  general  a  class  of  borders,  then  we  should  be  able  to  extend  the  methodology 
to  cover  well-defined  subclasses  of  non-linear  functions.  However,  If  the  problem  is 
caused  by  some  basic  characteristic  of  non-linear  borders,  we  will  not  be  able  to  extend 
doauin  testing  to  any  class  of  non-linear  functions. 

For  linear  borders,  we  have  assumed  that  if  the  given  border  Is  linear,  and  If  there 
Is  a  domain  error,  then  the  correct  border  Is  also  linear,  tn  order  to  extend  our 
testing  results  to  particular  subclasses  of  non -l inear  functions,  such  as  qusdrstic  or 
cubic  polynomisls,  we  must  assume  that  If  the  given  non-linear  border  it  in  error,  then 
the  correct  border  it  in  the  same  nan-linear  claas.  Th  1  a  non-linear  class  will  be  speci¬ 
fied  by  X  parameters:  for  sxample,  consider  the  general  form  of  a  two-dimensional 
quadratic  in  tares  of  variables  X  and  Y,  where  A,  9.  C,...  are  coefficients,  and  X  •  8i 

AX*  ♦  it'  •  CXT  ♦  OX  ♦  XT  ♦  F  •  0. 

Then  (R-l)  points  can  be  chosen  in  order  to  solve  for  these  X  coefficients.  For  the 

example  above,  the  five  points  C*1,  T^l,  l  •  l . S,  should  satisfy  the  following 

system  of  equations: 
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define  an  independent  ict  of  (K-l)  point*  (X  ,  Y^l  a*  a  aat  which  can  b*  a  a  ad  to  solve 
for  tna  coaf  f lclanta ,  and  thu*  determine  a  (pacific  raambar  of  tna  non- linaar  claaa. 


wa  can  now  formulate  tha  general  non-linaar  domain  tasting  strategy  In  tana  of  tnaaa 
observation*.  (K-l)  ON-orr  pair*  of  pom.*  ara  choaan  such  that  tna  (K-l)  ON  points 
ara  indapandant  and  aach  orr  point  1*  chosan  a  diatanca  (  from  tha  corresponding  ON 
point.  This  requires  2»(K-1)  taat  point*  par  non-linaar  bordar.  Tha  (K-l)  ON-orr  lina 
segments  formed  by  thl*  sat  of  pair*  hava  baan  chosan  so  that  tha  only  corract  bordar* 
which  yiald  corract  tast  rasult*  must  lntarsact  aach  of  tha**  ON-OFT  lln*  segment*. 

Tor  any  particular  corract  bordar,  thar*  ara  (K-l)  indapandant  intarsaction  points, 
wnicn  determines  tha  bordar  completely.  Not*  that  tha  intarsaction  point*  ara  indapand¬ 
ant  if  i  l*  cnosan  sufflciantly  small,  sine*  tha  ON  points  ara  indapandant  for  tha  glvan 
bordar,  A  furtnar  requirement,  as  in  tha  llnaar  ca*a,  is  that  all  OFF  point*  satisfy 
all  inequality  bordar*  othar  than  tha  on*  being  tastad. 


Whl la  a  single  OFF  point  wa*  sufflclant  in  tha  llnaar  csss,  tna  indapandanc*  criterion 
requires  (K-l)  OFT  point*  for  aacn  non-linaar  bordar.  In  tha  former  case  linearity 
allowed  tha  or>'  point  to  b*  shared  by  all  tha  ON  points,  sine*  tha  llnaar  independence 
of  tna  points  identified  as  lying  on  the  true  bordar  is  guaranteed  by  tna  llnaar  Inda¬ 
pandanc*  of  tna  ON  point*  themselves.  If  w*  war*  to  tast  a  non-linaar  bordar  with  (K-l) 
on  points  and  a  single  OFF  point,  w*  would  ba  able  to  conclude  that  tha  corract  and 
given  borders  lntarsact  at  (K-l)  point*.  However,  wa  cannot  conclude  that  these  (K-l) 
points  are  Indapandant.  w*  snow  of  no  s* (action  criterion  for  tha  ON  point*  whlcn 
would  guarantee  tha  indapandanc*  of  the  intersection  points  using  only  on*  OFT  point. 

So  an  affactlv*  strategy  requires  tha  full  sat  of  2*K  tast  points,  and  unfortunately  K 
grows  vary  rapidly  as  tha  dimensionality  and  dagraa  of  non-linearity  of  tha  bordar  in¬ 
crease*. 


A  two-dimensional  non- llnaar  bordar  is  s  vary  special  case,  and  even  though  tha  general 
strategy  is  affective,  a  slightly  different  tasting  strategy  can  o*  formulated  to  re¬ 
duce  tha  number  of  required  tast  point*.  Tha  basic  difference  1*  that  tha  intersection 
between  two-dimensional  non-linaar  border*  from  tha  same  class  is  a  finite  sat  of  point*, 
tna  maximum  number  of  whlcn  can  ba  determined  from  the  form  of  tha  function.  For  ex¬ 
ample  a  pair  of  two-dlmanslonal  quadratic  curve*  can  intersect  in  at  moat  four  point*. 
Thl*  mean*  that  any  sat  of  sure  than  four  points  cannot  possibly  11*  on  two  distinct 
quadratics,  and  any  five  point*  uniquely  determines  a  specific  quadratic.  Therefor*, 
wa  do  not  have  to  worry  about  indapandanc*  In  the  two-dlmanslonal  case,  sine*  any  *at 
of  (K-l)  distinct  point*  will  produce  *  system  of  indapandant  llnaar  aquation*.  For 
example,  any  three  distinct  point*  can  lie  on  at  most  ona  circle,  sine*  two  circle* 
cannot  hava  more  than  two  point*  in  common. 

Wa  tast  •  two-dlmanslonal  non-linaar  bordar  with  X  points,  a  g,  six  for  a  quadratic 
•election  In  an  ON -OFT -ON-OFF.  .. .  sequence  along  tha  bordar  *■  diagramed  for  tha  closed 
bordar  in  Figure  14.  Sine*  the  corract  bordar  must  pass  on  or  above  the  given  bordar 
at  aach  ON  point,  and  must  pas*  below  each  OFF  point,  tha  two  border*  must  lntarsact 


352 


s 

/ 

/ 

/ 

/ 

Cl  **n  border  _______ 

Correct  border  _____ 

fifure  14i  twatine  a  two- Jamana  1  one  1  non- llft»er  border 

on  odd  number  of  times,  let  us  assume  once,  in  each  ON-OFF  and  OFF-ON  Interval  along 
the  border.  The  K  test  points  define  (K-l)  Intervals  on  the  border,  each  of  which 
must  contain  at  least  one  Intersection  point.  Me  have  shown  that  these  (K-l)  points 
must  be  Independent,  and  since  they  cannot  lie  on  two  distinct  borders,  the  given 
border  must  be  correct  within  c.  ha  a  technical  detail.  It  la  also  possible  that  the 
correct  border  may  be  tangent  to  the  given  border  at  an  ON  point,  but  if  this  occurs, 
an  argument  Involving  the  derivatives  of  the  two  borders  at  that  point  can  be  Invoked 
to  Justify  the  choice  of  the  test  points  for  this  two-dimensional  case. 

Although  the  domain  strategy  has  been  extended  to  non-linear  boundaries,  points  must  be 
generated  in  a  domain  defined  by  non-llnear  boundaries,  requiring  the  solution  of  non- 
llnesr  systems  of  equations.  Since  this  probably  requires  excessive  processing  for 
arbitrary  non-llnear  borders.  It  does  not  represent  a  very  practical  approach. 


Adjacent  domains  which  compute  the  same  function 

If  two  adjacent  domains  compute  the  sane  function,  any  test  point  selected  for  their 
common  border  Is  ineffective,  since  the  same  output  values  are  computed  for  the  test 
point  regardless  of  the  domain  in  which  It  lies,  we  will  demonstrate  how  domain  test¬ 
ing  can  be  modified  to  deal  with  this  problem. 

In  Figure  IS (a),  assuming  domain  Di  were  being  tested,  we  must  compare  the  functions 
calculated  In  domains  D|  and  Dt  for  test  point  A,  0,  and  0.  for  B.  and  Di  and  Dt  for  C. 
One  of  the  major  problems  to  be  solved  is  the  identification  of  these  adjacent  domains. 
Me  assusie  that  when  testing  domain  0,  the  partitioning  structure  of  the  adjacent  doaulns 
and  the  program  paths  sssociated  with  these  domains  la  not  known.  It  would  be  very 
coaipllcated  to  have  to  generate  the  domains  which  are  adjacent  to  the  border  being  tes¬ 
ted. 


Figure  15(b)  Illustrates  an  approach  to  this  problem.  The  border  being  tested  Is  shif¬ 
ted  parallel  by  a  small  distance  t,  so  that  teat  points  A  and  B  now  belong  to  adjacent 
domains,  Dj  and  0.,  respectively.  The  modified  program  Is  than  retested  using  test 
points  A  and  B,  which  will,  as  a  by-product.  Identify  the  paths  associated  with  these 
two  adjacent  doaiains.  Me  can  then  compare  the  output  for  each  test  point  before  and 
after  the  shift.  If  It  Is  different,  then  we  can  definitely  conclude  that  the  adjacent 
domain  computes  a  different  function,  and  this  test  point  can  safely  be  used.  If  the 
output  Is  the  same  for  that  test  point,  then  we  can  conclude  that  either  assumption  1 
or  4  (page  })9)  la  violated.  However,  there  Is  no  way  to  decide  this,  and  the  only 
practical  approach  is  to  use  further  test  points.  If  we  know  that  coincidental  corr¬ 
ectness  cannot  occur,  then  we  could  conclude  on  the  basis  of  a  single  point  that  the 
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adjacent  domain  computaa  tha  sama  function. 

If  two  adjacant  dcmalna  auch  aa  0,  and  0t  In  Flgura  IS  (a)  ara  found  to  computa  tha  a ana 
function,  than  in  ordar  to  carry  out  tha  domain  taatlng  atratagy  on  tha  9 Ivan  bordar, 
naw  taat  polnta  nay  hava  to  ba  aalactad.  Por  axampla,  point  A  can  no  lon9ar  ba  uaad, 
and  thla  raquiraa  aacartalnlng  tha  bordar  atructura  batwaan  0,  and  Dt.  Thua  a  conald- 
arabla  amount  of  procaaaln9  la  raqulrad  which  la  probably  not  practical. 

In  aunamry,  a  tachnlqua  of  taatln9  aach  point  tv  lea  will  aaaura  ua  that  aaauaiptlon  4 
la  valid,  and  thla  radundancy  night  ba  vlawad  aa  a  raaaonabla  prlca  to  pay  to  allnlnata 
thla  raatrlctlon.  Howavar,  if  an  lnatanca  la  found  «diara  tha  aaaunptlon  la  not  valid, 
a  baalc  thaoratical  problan  axlata. 


Domain  testing  for  compound  predlcata  I 

Assumption  )  (paga  JJ»)  ststsd  that  s  pstn  contained  only  simple  prsdicstss,  snd  this 
Implied  thst  ths  sst  of  input  points  could  be  characterized  quits  simply  ss  s  single 
i  mein.  Ms  must  consider  whet  cosipllcstlons  csn  occur  for  compound  predlcstes,  snd  how 
the  dome  in  strategy  csn  be  general lied  to  test  paths  containing  these  predlcstes. 

The  set  of  Inputs  corresponding  to  a  path  is  defined  by  the  path  condition,  consisting 
of  the  conjunction  of  the  predicates  encountered  slong  the  path.  If  a  compound  predi¬ 
cate  of  the  fora  ICIll  AND  011*1)1  is  encountersd  on  the  path,  the  path  condition  le 
still  a  single  conjunction  of  simple  predicates,  and  the  only  difference  is  that  two 
of  the  simple  predlcstes  are  produced  as  a  single  branch  point  on  the  path.  No  modi¬ 
fications  of  the  domain  testing  strstegy  are  required  in  this  esse. 

However,  compound  predicates  using  the  Booleen  operstor  OR  sre  more  complicated.  Con¬ 
sider  s  pstn  contslning  the  following  predlcstes i 

Ci.  C i ,  ....  I C4  ^1*1^*  ...  ct 

The  pstn  condition  in  this  case  is  the  conjunction  of  these  predicates,  and  in  standard 
disjunctive  normal  formi 


(C,  AsO  C|  AND  ...  ANO  Ct  AND  ...  AND 

OR  (C ,  ANO  C,  AND  ...  AnO  C4>1  ANO  ...  ANO  Ctl 

The  set  of  Input  date  points  following  this  path  consists  of  the  union  of  two  domains, 
esch  defined  by  the  conjunction  of  simple  predicates,  and  in  general  any  number  of  these 
domains  are  possible. 

Assuming  linear  predicates,  each  of  these  domains  is  a  convex  polyhedron,  but  the  do¬ 
mains  may  overlap  in  arbitrary  ways.  The  major  problem  caused  by  these  compound  predi¬ 
cates  Is  that  the  domains  correspond  to  the  same  path,  and  the  assumption  that  adjacent 
domains  do  not  compute  the  same  function  is  violsted.  Me  identify  these  cases  of  im¬ 
portance:  domains  which  do  not  overlsp,  domains  which  partlslly  overlap,  and  domains 

wnich  totally  overlap. 

The  first  case  is  indlcsted  In  Figure  ltla),  where  domslns  D,  and  0t  are  defined  by  tne 
compound  predicate  (Ct  OR  C,  ) ,  and  domain  3,  corresponds  to  son  other  psth.  In  this 
csss  our  methodology  can  be  applied  to  each  domain  separately,  since  tne  two  domains 
for  this  path  are  not  adjacent. 

In  Tlgure  lt(b),  the  doeialns  partially  overlap,  where  D|  u  Di  le  the  doauln  defined  by 
C i  ,  and  D,  \j  Di  is  the  domain  defined  by  Ci.  In  the  exas^le  we  cannot  test  the  domains 
separately,  since  they  are  adjacent  and  the  tan  function  is  coeiputed  in  eacn.  For 
example,  any  test  point  for  C(,  selected  along  that  part  of  the  border  between  D|  and 
U i.  Is  Ineffective  since  the  tan  results  are  computed  for  it  in  both  of  those  regions. 
So,  in  this  case  we  must  ensure  that  the  adjacent  domain  assumption  is  satisfied  by 
selecting  test  points  for  C,  and  Ci  which  lie  in  that  part  of  the  border  adjacent  to  a 
domain  for  son  other  path.  In  order  to  deal  effectively  with  this  ease,  son  extra 
analysis  will  have  to  be  made,  first  In  order  to  Identify  this  second  ease,  and  also 
to  Identify  the  actual  domain,  which  is  no  longer  convex.  The  borders  of  this  domain 
are  thorn  with  a  heavier  line  in  Figure  lt<b).  This  is  probably  no  longer  a  practical 
approach,  especially  for  higher  dlnnslons. 
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The  third  caaa  la  shown  In  figure  U(c)  ,  whara  tha  domain  D,  for  pradlcata  C,  la  a 
aubaat  of  tha  othar  domain,  0,  u  D, ,  which  la  obtalnad  for  pradlcata  C, .  Thla  praa- 
anta  a  aarioua  problam  ainca  thara  ara  no  taat  potnta  for  bordar  B  of  domain  Di  which 
can  aatlafy  tha  ad)acant  domain  assumption,  and  tharafora  B  cannot  ba  taatad  affectively 
Tha  technique  davalopad  In  tha  pravioua  aactlon  should  halp  to  ldantlfy  thla  caaa. 
Howavar,  avan  If  thla  caaa  could  ba  idantlfiad.  testing  for  bordar  B  la  no  longer  a 
practical  procadura. 

So,  In  aummary,  a  compound  pradlcata  of  tha  form  tCl  UNO  C21  la  tha  aama  aa  two  almpla 
predicate*.  and  doaiin  testing  can  ba  appliad  to  a  domain  daflnad  with  thla  typa  of 
compound  pradlcata.  In  addition,  If  tha  compound  pradlcata  la  of  tha  form  (C,  OR  C,  1 
and  tha  domalna  ara  diatinct,  domain  tearing  can  ba  appliad  to  aach  domain  aaparataly. 
Howavar,  If  tha  domalna  ovarlap,  thla  introducaa  tha  problam  of  adjacent  domalna  which 
camputa  tha  aama  function.  Although  wa  may  not  find  affactlva  taat  polnta  for  domalna 
which  ovarlap  in  arbitrary  waya,  wa  can  racoqnue  thla  altuatlon  and  ldantlfy  It  aa  a 
bordar  which  cannot  ba  taatad  affactlvaly. 


ERROR  ANALYSIS  or  DOMAIN  BORDERS  AND  PISCRCTE  SPACES 

An  arror  analyala  of  domain  bordara  la  naadad  to  raaolva  tha  following  questions: 

1  How  amall  ahould  c  ba  choaan  In  selecting  an  orr  taat  point  for  llnaar  bordara,  and 
whara  ara  optimal  location*  for  tha  taat  polnta? 

2  Wa  required  tha  orr  taat  point  for  a  g  1  van  bordar  to  aatlafy  all  Inequality  bordara 
axcapt  that  being  taatad:  how  do  potantlal  arror*  In  othar  bordara  of  tha  domain 
affact  thla  requirement? 

)  What  ara  tha  difflcultl**  In  applylnq  domain  testlnq  In  a  dlacraat  apaca  or  In  a 
apaca  in  which  numerical  valuaa  can  only  ba  ropraaantad  with  flmta  raaolutlon,  and 
can  thaaa  dlfflcultlaa  ba  circumvantad  by  taklnq  raaaonabla  precaution*  with  tha 
aathod? 

Thaaa  and  othar  arror  analyala  problama  ara  daalt  with  In  datall  In  (012).  It  la  Inter¬ 
esting  to  obaarva  that  tha  anawara  to  questions  1,  2  and  ]  all  Involve  tha  aama  worat- 
caaa  altuatlon:  whan  two  adjacent  llnaar  bordara  of  tha  aama  domain  ara  nearly  parallel, 
figure  1?  indicate*  tha  two  caaaa  which  can  arlaa  from  adjacent  llnaar  bordara  which 
ara  nearly  parallel.  rigura  17(a)  ahowa  a  given  bordar  segment  ZT  In  which  tha  two 
adjacent  bordar  segments  EP  and  TO  both  make  large  external  angle*  a,  and  8j ,  near  180*, 
with  tha  given  bordar  EP.  Thla  laada  to  vary  amall  aupplementary  internal  angle*  9, 
and  9t.  and  aapacially  for  9>  ,  thla  raaulta  in  a  vary  aharp  ’ corner '  of  tha  domain.  In 
figure  17(b),  tha  adjacent  bordara  PE  and  P0  are  again  nearly  parallel  to  tha  given 
bordar  IP,  but  a  different  caaa  la  created.  In  thla  caaa,  external  anglaa  •,  and  8t 
ara  very  amall,  and  tha  internal  anglaa  9,  and  9i  ara  both  near  180*. 

Wo  will  briefly  argue  In  thla  paper  that  one  of  tjiese  two  altuatlono  la  tha  key  to  tha 
analyala  of  quaatlona  1,  2,  and  J,  and  wa  refer  the  reader  to  (01?)  for  further  datalla 
and  proofr.  Tha  beat  location  for  aach  of  tha  three  taat  polnta  will  bo  Indicated,  and 
It  will  ba  ahown  how  interacting  bordar  changaa  may  affact  tha  location  of  thaaa  taat 
polnta.  Tha  problem  of  doauin  taatlng  In  dlacraat  apacaa  la  briefly  Introduced,  and  a 
aufflelent  condition  la  apoclflod  which  guarantaaa  that  affactlva  taat  potnta  can  alwaya 
ba  choaan.  Since  all  thaaa  argument*  ara  given  only  for  two  dUaanalona,  a  brief  dlacuaalon 
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will  d«4l  with  th«  qmnmr* U *«t ion*  to  hl?h«r  dimension*. 

An  arror  nataura  for  taat  point  aalactlon 
• 

In  Plqura  17(a),  conaldar  tha  aalactlon  of  thraa  taat  pointa  A,  B.  and  c  for  taatlnq 
bordar  aaqaant  ZT .  It  la  ahown  in  (012)  that  tha  baat  poaltlona  for  two  of  thaai,  aay 
A  and  B.  ara  pointa  E  and  r,  ao  tha  raaialnlnq  problem  la  tha  location  of  taat  point  C. 
Ma  hava  obaarvad  that  if  tha  qlvan  bordar  E r  la  in  arror,  than  taat  pointa  A,  B  and  C 
will  fall  to  datact  arrora  If  tha  corract  bordar  la  ona  which  lntaraacta  llna  aaqmanta 
AC  and  BC.  Thua  q 1 van  C  which  la  at  a  dlatanca  c  froa  tha  qlvan  bordar  and  halfway 
batwaan  A  and  B,  an  approprlata  arror  crltanon  could  ba  tha  'number*  of  arronaoua 
pointa  which  would  ba  undatactad,  1  a,  tha  area  between  tha  two  border*,  poaalbly 
Hal  tad  by  althar  or  both  of  tha  axtenalona  of  tha  adjacent  bordara  EP  and  rQ.  It  can 
ba  ahown  that  thla  araa  aaaaura  can  ba  boundad  by  tha  axpraaalon 

_ ilUl' _ 

IE  ♦  2c  cot  •  , 

whara  •  la  tha  larqar  of  and  0a . 

In  ordar  for  thla  arror  aaaaura  to  ba  finite,  It  la  nacaaaary  that  both  0,  and  •>  ara 
not  too  cloaa  to  1BO*  for  qlvan  c.  If  |  cot  •  |  <<  ^  ,  than  tha  arror  aiaaaura  la  In 
tha  ordar  of  cEf.  Thla  qivaa  ao*»e  quidance  aa  to  tha  cholca  of  t  for  point  C. 


Infracting  border  segments 


In  presenting  tha  domain  stratagy,  we  required  the  orr  taat  point  to  satisfy  all  in¬ 
equality  bordars  axcapt  tha  bordar  being  tastad.  Usually  this  doas  not  Inposa  much  of 
a  constraint  on  tha  cholca  of  tha  OFF  point,  but  riqura  17(b)  lllustratas  a  situation 
In  which  a  savors  constraint  axlsts.  Wa  can  show  that 


(cot  S|  *  cot  9| i  , 

and  sines  c  <  h  for  chooalng  tha  orr  taat  point,  this  again  shows  tha  affact  If  althar 
#i  or  0,  or  both  aro  vary  small. 

Tha  same  situation  applies  for  Interacting  adjacent  bordars,  and  Is  Illustrated  In  Flg- 
ura  II  .  As  long  as  tha  orr  points  Ci  and  C|  for  each  of  tha  adjacent  bordars  are  chosen 
sufficiently  close  to  those  borders,  and  tha  external  angles  0!  and  0,  aro  not  too  small, 
than  tha  adjacent  borders  have  a  minimal  Influence  on  tha  selection  of  tha  orr  point  C  for 
bordar  Er.  Tor  example,  point  C  must  llo  Inside  triangle  EHJ  determined  by  given  borders 
tf  and  rQ.  Tha  correct  bordars  which  pose  tha  worst  case  In  limiting  tha  selection  of 
point  C  are  shown  as  dashed  Unas  in  Figure  18>  these  limiting  correct  bordars  are 
determined  by  how  close  Ct  and  Ct  have  been  chosen  to  their  respective  tost  bordars. 

As  a  result  of  those  conditions,  point  C  Is  constrained  to  lie  within  triangle  EFV .  a 
more  restrictive  condition  than  presented  by  triangle  EFU.  It  should  be  clear  that  if 
either  0,  or  0i  Is  too  small,  or  either  C,  or  Cs  is  chosen  too  far  from  Its  respective 
test  border,  the  region  from  which  C  could  be  chosen  would  become  restrict lvely  small. 


Possible  correct  borders 


Flfere  If,  effect  of  interacting  adjacent  borders  on  teat  point  C 
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Dlicr»»t  space  analysis 


The  previous  several  (actions  have  indicated  that  if  adjacent  borders  ara  nearly  par* 
allal,  than  tast  point  C  Is  required  to  lie  vary  closa  to  tha  bordar  belnq  tastad.  But 
In  a  dlscraat  spaca  this  could  causa  a  savara  problem,  for  no  dlscraat  point  nay  exist 
that  closa  to  tha  bordar.  Similar  problems  axlst  for  tha  ON  tast  points  A  and  B,  for 
It  may  not  ba  posslbla  to  choosa  than  at  axtrama  points  of  tha  bordar. 

Tor  tha  dlscraat  spaca  ■«  shall  consldar  a  two-dimensional  lattlca,  with  uniform  spac* 
mq  1  In  both  dimansiona.  This  nodals  tha  situation  whara  tha  Sana  data  raprasantatlon , 
Inteqer  or  flxad-point,  is  uaad  for  two  input  varlablas. 

T or  simplicity,  lat  us  aqaln  assume  that  points  A  and  B  can  ba  chosen  as  points  C  and 
r.  We  shall  present  a  sufficient  condition  for  a  qlven  domain  within  this  dlscraat 
lattlca  which  quarantees  that  an  OFT  point  C  can  ba  chosen  as  a  lattice  point  for  each 
bordar  so  that  the  area  criterion  is  finite.  Tha  result  is  based  upon  tha  followlnq 
two  observations.  FTrst,  any  circle  of  diameter  .7  i  always  contains  at  least  one 
lattlca  point.  Second,  from  riqure  17(a),  note  that  if  either  external  anqles  B,  or  B| 
ara  too  near  ISO*,  than  the  'width'  of  tha  domain  will  tend  to  be  vary  small  In  terms 
of  tha  lattlca  resolution  i. 

Nora  formally,  define  tha  <*i, eater  d  of  a  convex  polyqonal  domain  to  be  tha  shortest 
distance  from  any  extrema  point  to  any  domain  edqe  not  adjacent  to  that  extrema  point) 
this  corresponds  to  our  informal  arqument  about  domain  'width'.  The  sufficient  con¬ 
dition  can  then  ba  stated  asi 

Fraiotllloi  S 

Given  a  domain  with  diameter  d  In  a  lattice  with  resolution  A,  If 

<1  »  J  —  1  i  -  (2*12)  d, 

U/ 

th«n  *  lattice  OFF  point  can  be  cboaen  for  every  border,  and  moreover  all  external 
angles  6t  and  are  constrained  by 

|  cot  6,  ♦  cot  Gj  {  < 

It  Is  clear  that  there  ara  soma  domains  In  a  dlscraat  spaca  which  cannot  ba  tastad,  but 
these  ara  patholoqlcal  cases  whara  one  of  tha  domain  dimensions  Is  in  tha  order  of  tha 
iattlc  resolution.  Moreover,  tha  result  Indicates  a  simple  computation  in  terms  of  the 
domain  diameter  to  determine  when  such  domains  are  presented  for  testlnq.  Tor  domains 
which  can  ba  tested  In  a  discreet  spaca,  tha  important  result  from  Proposition  5  Is  that 
s  restriction  has  bean  obtained  on  anqles  Bi  and  •  which  precludes  both  anqles  which 
srs  closa  to  ISO*  and  anqles  which  ara  too  small. 


Extensions  of  error  analysis  to  higher  distensions 

Tha  previous  arguments  hava  all  bean  made  for  two  dimensions,  so  it  la  Important  that 
tha  essential  ideas  can  be  qenarallzad  to  hiqher  dimensions.  We  can  observe  that  If 
two  bordar  aeqments  era  adjacent,  they  ara  lntarsectlnq  hyparplanas.  Aqain,  problems 
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My  cut  It  these  two  hyperplanes  H,  and  H,  are  nearly  parallel,  and  this  can  be  meas¬ 
ured  by  taking  the  inner  product  of  their  unit  normal  vectors  h,  and  rtt,  yielding  the 
cosine  of  the  angle  •  between  them 


cos  «  •  rti  •(!, 

This  calculation  is  straightforward,  and  all  the  other  error  analysis  issues  can  be 
extended  in  a  similar  way.  The  reader  should  examine  (01?)  for  further  details. 


CONCLUSIONS  AND  ftTUBg  WORK 

The  basic  goal  of  this  resesrch  Is  to  replace  the  intuitive  principles  behind  current 
testlnq  procedures  by  a  methodology  based  on  a  formal  treatment  of  the  program  testlnq 
problem.  >y  formulating  the  problem  In  basic  geometric  and  algebraic  terms,  we  hsve 
been  able  to  develop  an  effective  testing  methodology  whose  capabilities  can  be  precisely 
defined.  In  addition,  since  program  testing  cannot  be  completely  effective,  we  have 
Identified  the  limitations  of  the  strategy.  In  several  cases  these  limitations  have 
proven  to  be  theoretical  problems  inherent  to  the  general  program  testing  process. 

The  Min  contribution  of  this  research  la  the  development  of  the  domain  testing  strat¬ 
egy.  Under  certain  well-defined  conditions  the  methodology  la  guaranteed  to  detect 
doneln  errors  In  linear  borders  greater  than  some  small  magnitude  c.  Furthermore,  the 
cost,  as  measured  by  the  number  of  required  test  points,  is  reasonable  and  grows  only 
linearly  with  both  the  dimensionality  of  the  input  space  domain  and  the  number  of  path 
predicates.  Domain  testing  also  detects  transformation  errors  and  missing  path  errors 
In  many  cases,  but  the  detection  of  these  two  classes  of  errors  cannot  be  guaranteed. 

Doeialn  testing  has  also  been  extended  to  classes  of  non-linear  borders,  and  we  have  shown 
that  the  methodology  generalises  to  any  class  of  functions  which  can  be  described  by 
a  finite  number  of  parameters.  Unfortunately,  non-linear  predicates  pose  problems  of 
extra  processing  which  probably  preclude  testing  except  for  restricted  cases.  For 
example.  Just  finding  intersection  points  of  a  set  of  linear  and  non-linear  borders  can 
require  an  inordinate  amount  of  processing. 

Coincidental  correctness  is  a  theoretical  limitation  inherent  to  the  prograft  testing 
process,  and  we  have  argued  that  it  prevents  any  reasonable  finite  testing  procedure 
from  being  completely  reliable.  In  particular,  the  possibility  of  coincidental  correct¬ 
ness  means  that  an  exhaustive  test  of  all  points  in  an  Input  domain  la  theoretically 
required  to  preclude  the  existence  of  confutation  errors  on  a  path.  within  the  class 
of  all  computable  functions  there  exist  functions  which  coincide  at  an  arbitrarily  large 
nussber  of  points,  but  If  there  Is  sufficient  resolution  in  the  output  space,  coincid¬ 
ental  correctness  should  be  a  rare  occurrence  for  functions  commonly  encountered  In  data 
processing  problems. 

The  class  of  missing  path  errors,  particularly  those  of  reduced  dimensionality,  has 
proven  to  be  another  theoretical  limitation  to  the  reliability  of  any  finite  testing 
strategy.  Although  our  methodology  cannot  be  guaranteed  to  detect  all  Instances  of  this 
type  of  error.  It  can  be  extended  to  detect  soa<e  well-defined  subclasses  of  missing  path 
errors.  Unfortunately,  the  extra  cost  of  this  modification  My  be  unacceptably  high. 

Our  analysis  of  missing  path  errors  has  shown  that  the  cause  of  the  difficulty  is  that 
the  program  does  not  contain  any  indication  of  the  possible  existence  of  a  missing  path 
error.  Therefore,  without  additional  lnfonaatlon,  a  reasonable  testing  strategy  for 


this  class  of  srrors  cannot  ba  formulated. 


The  domain  testing  strategy  requires  a  reasonable  number  of  test  points  for  a  single 
path,  but  the  total  cost  may  be  unacceptable  for  a  large  program  containing  an  excess¬ 
ive  number  of  paths.  In  particular,  this  may  occur  for  large  programs  with  complicated 
control  structures  containing  many  iteration  loops.  Additional  research  Is  needed  to 
reduce  substantially  the  number  of  potential  paths.  One  area  being  investigated  is  to 
obtain  appropriate  restrictions  on  control  structures,  especially  Iteration  loops,  so 
that  a  large  number  of  paths  can  be  tested  simultaneously  for  domain  errors.  Another 
approach  is  to  develop  an  objective  criterion  to  select  a  small  subset  of  paths  which 
yields  the  greatest  testing  Information.  A  figure  of  merit  must  be  defined  which  re¬ 
flects  the  value  of  a  path  as  a  candidate  for  testing,  and  this  measure  must  consider 
both  the  benefits  and  the  cost  of  testing  each  path.  For  example,  a  very  long  and  com¬ 
plicated  path  containing  many  assignment  statements  and  predicates  can  test  many  aspects 
of  the  program,  but  a  larger  number  of  test  points  are  consequently  required.  There¬ 
fore,  the  best  candidate  for  testing  might  be  a  long  path  consisting  of  many  assignment 
stateiaents  but  few  predicates.  Also,  in  selecting  the  next  path  for  testing  we  must 
consider  how  the  set  of  paths  already  chosen  affects  our  current  selection.  Me  have 
seen  that  a  single  error  may  affect  many  different  paths,  and  the  error  can  be  detected 
by  testing  any  one  of  these  paths.  A  third  approach  to  reduce  the  number  of  paths  is 
to  design  and  test  small  program  modules,  and  then  construct  a  testing  strategy  for  the 
combination  of  these  nodules  Into  large  software  systems.  This  is  the  most  practical 
solution  to  the  testing  problem,  but  many  technical  and  theoretical  problems  need  to 
be  resolved.  For  example,  in  order  to  accomplish  this,  the  communication  between  mod¬ 
ules  will  have  to  be  appropriately  restricted. 

Me  have  assusied  that  an  'oracle'  exists  which  can  always  determine  whether  a  specific 
test  case  has  been  computed  correctly  or  not.  In  reality,  the  programmer  himself  must 
make  this  determination,  and  the  time  spent  examining  and  analysing  these  test  cases  is 
a  major  factor  In  the  high  cost  of  software  development.  One  possible  avenue  for  future 
research  would  be  to  automate  this  process  by  using  some  form  of  input/output  specific¬ 
ation.  If  the  user  provides  a  formal  description  of  the  expected  results,  the  correct¬ 
ness  of  each  test  case  can  be  decided  automatically  by  determining  whether  the  output 
specification  Is  satisfied.  This  would  reduce  the  cost  of  testing  tremendously,  and 
these  new  testing  techniques  would  gain  acceptance  more  quickly  since  the  tedious  task 
of  verifying  test  data  would  be  eliminated.  In  addition,  any  extra  information  supplied 
by  the  user  might  be  useful  In  specifying  special  processing  requirements  which  would 
Indicate  the  existence  of  a  possible  missing  path  error. 

The  doaialn  teat  strategy  Is  currently  being  implemented,  and  will  be  utilised  as  an 
experimental  facility  for  subsequent  research.  Experiments  should  indicate  what  sort 
of  proqraanlnq  errors  are  most  difficult  to  detect,  and  should  yield  extensive  dynamic 
testing  data.  A  most  important  contribution  would  be  to  Indicate  both  programing 
language  constructs  and  programming  techniques  which  are  easier  to  test,  and  thus  would 
produce  more  reliable  software. 
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Discussion 


A  Discussion  o f 

A  SURVEY  OF  PROCRAM  TESTING  ISSUES 

Lee  J  While.  Edward  I  Cohen,  and  B  Chandrasekaran 
Ohio  Slate  University.  Columbus 


In  the  survey  paper  on  proeram  testing.  John  Goodenough  identifies  a 
number  of  dillereni  possihle  objectives  for  testing  He  indicate)  that  with  each 
objective  the  approach  to  testing  should  be  different  The  moil  important  point 
emphasired  it  the  need  for  more  research  and  undemanding  of  the  banc  tewing 
phenomenon  in  order  that  better  technique!  can  be  designed  baled  upon  thii 
underlying  theory  Mu  lurvey  doei  not  include  tome  recent  theoretical  retulti 
which  give  cauie  for  optimum  that  a  theoretically  lound  program  testing 
methodology  will  ertirrre  We  would  thui  like  to  bring  hu  lurvey  up  to  date  by 
giving  a  brief  account  of  reiul'i  which  are  preiented  in  greater  detail  in  [1.21 

The  remit!  to  be  reported  have  at  their  objective  prefia"*  eotrttlnen. 
referring  to  the  Goodenough  paper  The  construction  and  correctness  of  a 
program  specification  M  t  difficult  re<»arch  problem  in  i*s  own  right  (as 
emphauted  in  this  conference)  Thus  the  paradigm  for  thii  research  will  assume 
that  the  program  structure  u  spectlird  that  test  points  are  to  be  selected  to  test 
the  program,  and  that  an  "oracle"  (or  user)  will  be  available  to  indicate  the 
correctness  or  incorrectnen  of  input  output  pairs  Furthermore,  the  approach 
will  be  that  of  pe/A  eriented  ce'recfseii  luting 

A  number  of  authors  (eg.  Howden  [?]>  have  identified  a  ifenem  error  as 
an  error  in  the  predicate  of  a  program  path  in  that  a  specific  input  will  follow 
the  wrong  path  due  to  an  error  in  the  control  flow  of  the  program.  The 
research  reported  in  this  discussion  will  obtain  a  finite  (and  reasonably  small) 
number  of  test  points  to  completely  test  for  a  domairi  error  along  a  specific 
program  path  The  remit  will  be  stated  and  suitably  qualified  below 

The  term  "domain  error*  arises  from  the  fact  that  the  set  of  distinct  paths 
in  the  program  partition  the  input  space  into  regions  or  "domains" 
Corresponding  to  each  feasible  path  there  are  one  or  more  domains  T'  * 
boundaries  of  each  domain  are  determined  by  predicates  in  the  con  1 
statements  along  that  path  Recognition  that  these  boundaries  are  typica 
/inrer  in  the  input  variables  has  allowed  other  investigators  (Clarke  (l)  and 
Boyer  et  al  [5D  to  utilite  linear  programming  techniques  to  obtain  test  points  A 
survey  of  fifty  typical  data  processing  programs  has  shown  that  virtually  all  of 
the  predicates  are  linear  for  this  class  of  programs  Although  our  original  work 
on  domain  resting  assumed  that  the  predicates  were  linear  in  the  input  variables, 
we  have  since  shown  [2]  that  the  technique  ratendi  to  certain  classes  of  nonlinear 
predicates  I 
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Ne*t  consider  several  problem  encoun'.errd  by  any  type  of  testing,  but  in 
particular  by  the  proposed  domain  testing  approach  The  first  problem  might 
be  termed  ■coincidental  correctneo".  where  although  a  test  point  it  following  the 
wrong  program  path  due  to  a  control  error,  it  coincidentally  yields  the  correct 
output  Tho  problem  will  prove  imurmountabie  for  any  teat  point  selection 
strategy,  and  the  only  remedy  is  redundant  test  point  selection  Thus  in  the 
domain  testing  result,  we  shall  assume  coincidental  correctness  does  not  occur,  as 
it  should  occur  very  tarely  for  practical  programs  Another  problem  is  that  of  a 
"lining  fwr*  errer  (also  called  a  luSjst  t'tgr  by  Howden  [)D.  in  which  a 
required  predicate  is  missing  from  the  program  altogether  This  becomes 
especially  acute  if  the  missing  predicate  is  of  an  equality  type  along  the  path,  for 
then  the  reduction  in  dimensionality  of  the  domain  makes  it  virtually  impossible 
to  detect  the  error  by  test  point  selection  No  test  point  selection  strategy  can 
overcome  problems  of  "unmg  path  men  of  rnfuceif  rfin/niena/iry.  although  the 
proposed  domain  testing  technique  can  be  modified  to  detect  other  types  of 
missing  path  errors 

The  domain  testing  result  can  now  be  stated 

Theorem  Consider  a  computer  program  with  linear  predicates  in  terms  of 
the  input  variables  Then  assuming  no  coincidental  correctness  of  any  test 
points,  and  evcept  for  missing  paih  errors,  a  convey  polytope  R  in  the  input 
space  coi responding  to  a  specific  path  in  the  program  can  be  tested  by  a  finite 
number  of  points  as  to  whether  a  predicate  etror  in  the  program  has  shifted  the 
boundaries  ot  R  Moreover,  the  number  of  test  points  required  is  no  more  than 
PN.  wheie  N  is  the  dimensionality  of  the  input  space  and  P  is  the  number  of 
piedicates  along  (he  path 

The  above  theorem  is  rather  significant  In  the  area  of  program  testing, 
even  eyrluding  coincidental  corre-.'ness  and  missing  path  errors,  in  no  other 
situation  do  we  have  resulis  which  give  assjrance  of  complete  testing  for  any 
son  of  erior  with  a  finite  number  of  test  points  Furthermore,  the  test  points 
selected  for  this  purpose  will  also  partially  test  the  program  for  other  types  of 
errors,  vu .  ceiputatton  t’ten.  as  identified  by  Howden  (31  Only  coincidental 
correctness  of  the  computational  function  along  the  path  would  mislead  one 
about  the  information  provided  by  such  points 

The  technique  hat  also  been  eitended  to  nonlinear  predicates  Just  as  in 
the  linear  case  described  above,  the  assumption  here  is  that  if  a  predicate  is 
found  to  be  in  error  (thus  leading  to  a  domain  change),  the  intended  or  "correct* 
predicate  is  also  in  the  same  nonlinear  class  Again  this  is  not  an  unreasonable 
assumption  in  most  practical  environments  However,  a  larger  number  of  tew 
points  would  have  to  be  selected,  depending  upon  the  form  of  nonlinearity 
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Thu  rwixh  thou  Id  provide  a  batit  for  path-oriented  letting,  but  a 
number  of  teriout  practical  problem!  remain  which  mutt  be  addreued  in  order 
to  deugn  a  practical  tett  generation  tyiiem  Path  telection  u  Mill  a  difficult 
problem,  and  an  eitention  of  (he  work  of  Howden  (6)  it  needed  Iteration  loops 
pretent  a  proliferation  of  unnecettary  patht  for  domain  letting,  here  U  where  a 
combination  of  verification  and  letting  technique!  may  provide  a  more  practical 
and  yet  tolidly  bated  approach  The  tuccetiful  retolution  of  thete  and  other 
problem!  by  rigor  out  retearch  thoukl  provide  tubttantial  tewing  guidance  for 
(he  deugn  of  Urge  toft  ware  tyttemt 
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T«st  tools:  ussfulnsss 
must  sxtsnd  to  svoryday 
programming  snvlronmsnt 

B.  ChandraMkaran 
Ohio  Stata  Urn  vanity 

Currant  software  tasting  tools  are 
designed  to  Help  tha  tasting  procass 
by  highlighting  diffarant  sourcas  of  er¬ 
ror  in  diffarant  ways.  Each  tool  imple¬ 
ments  a  particular  theory  or  approach 
behaved  bast  suited  for  uncovering 
certain  types  or  patterns  of  srrora.  In¬ 
creasing  evidence  indicates  that  no 
single  tool  will  suffice  to  test  a  com¬ 
plex  program.  A  wall-intagrated  cof¬ 
fee  non  of  tools,  with  sach  tool  ap¬ 
propriate  for  certain  error  types,  will 
be  needed  for  any  senous  testing  facili¬ 
ty.  This  raises  questions  of  research 
interest— which  combination  of  tools 
provides  the  best  coverage  across 
types  of  errors?  What  are  the  criteria 
for  integrating  the  tools  le.g..  some 
tools  might  provide  information 
which  may  be  used  as  input  to  others)*1 
How  do  we  evaluate  an  integrated 
facility  as  opposed  to  evaluating  in¬ 
dividual  tools? 

An  important  consideration  in  the 
usefulness  of  a  tool  is  the  degree  to 
which  it  may  be  incorporated  in  an 
average  programmer's  environment. 
Tools  for  data  flow  analysis,  for  in¬ 
stance.  are  easily  incorporated  in  such 
an  environment,  as  are  some  execution 
monitoring  instrumentation  facilities. 

Tools  are  also  language-dependent, 
i.s..  most  tools  are  designed  for  hand 
ling  programs  written  in  one  lan¬ 
guage.  Modulations  will  be  needed  to 
adapt  them  to  other  languages.  But 
more  important  is  the  relation  be¬ 
tween  tools  and  language  constructs. 
For  instance,  while  current  data  flow 
analysis  techniques  can  handle  most 
single-process  programs,  there  ta  a 
need  for  new  analytic  techniques  for 
dealing  with  concurrent-process  pro¬ 
grams.  The  synchronisation  con¬ 
structs  that  characterise  the  latter  in¬ 
troduce  complex  data  and  control  flow 
possibilities. 

It  is  useful  to  categorise  currently 
available  software  tools  into  three 
classes— static  dynamic,  and  inter 
pretive.  Static  analysis  tools  work  on 
the  structure  of  the  program  and  do 
not  involve  execution.  Facilities  for 
data  flow  analysis  and  for  gathering 
information  such  as  cross-reference 
maps  are  examples  of  this  type  of  tool 
Program  verification  is  also  a  static 
analysis  technique.  Examples  of 
dynamic  analysis  tools  include  path 


generation  routines  and  instrumenta¬ 
tion  for  execution  monitoring.  The 
results  of  dynamic  analysis  are  based 
on  the  performance  of  the  program  as 
it  is  executed  for  some  inputs.  On  the 
other  hand,  symbolic  execution— an 
interpretive  technique— "executes'' 
the  program,  but  not  for  real  inputs 
Instead  input  ia  in  symbolic  form  and 
the  execution  computes  symbolic 
values  for  program  and  output 
vans  bias. 

Workabop  preeentatioaa.  One  ses¬ 
sion  was  specifically  devoted  to  test 
tools,  while  other  workshop  papers 
contained  some  discussion  of  ex¬ 
periences  with  various  test  tools. 

A.  Amschler  from  Karlsruhe.  West 
Germany  ico-authors  were  L.  Gmetner 
and  U.  Vogesi.  presented  her  group's 
work  in  the  design  and  implementa¬ 
tion  of  an  integrated  testing  system 
called  SADAT  (presumably  an  acro¬ 
nym  for  Static  And  Dynamic  Analysis 
and  Testing)  for  testing  Fortran  pro¬ 
grams  that  have  been  compiled  error- 
free.  The  main  modules  of  SADAT  are 
static  analyzer,  dynamic  analyzer, 
test  case  generator,  and  path 
predicate  calculator.  The  static 
analyzer  produces  several  tables 
based  on  a  simplified  lexical  analysis 
of  the  program  source  code  and  also 
generates  a  reduced  program  graph 
Several  types  of  errors  can  be  detected 
at  this  stage,  such  as  dead  code, 
undeclared  or  unused  labels  and 
variables,  and  jumps  into  a  loop  Inad- 
dition.  the  output  of  the  static 
analysis  phase  serves  as  a  data  base 
for  later  analysis. 

SADAT's  dynamic  analysis  docu¬ 
ments  the  execution  of  program  test 
runs  Basically  this  consists  of  instru¬ 
mentation  for  the  execution  count  of 
various  branch  points.  A  table  ia 
printed  giving  the  relative  and  ab¬ 
solute  numbers  of  executions  and 
identifying  those  paths  not  executed 
dunng  the  test  runs.  This  dynamic 
analysis  is  useful  for  identification  of 
dead  code,  determining  correctness  of 
loop  iterations,  and  optimization 

The  test  generation  subsystem 
automatically  generates  a  subset  of 
paths  with  almost  complete  C. -cov¬ 
erage  (i  s.,  each  arc  and  each  node'  is  re¬ 
presented  in  at  least  one  pathl.  In  addi¬ 
tion  to  the  automatically  generated 
paths,  the  user  can  specify  a  path  as  a 
sequence  of  statements. 

The  final  module,  not  yet  fully  im¬ 
plemented.  calculates  path  predicates 
by  symbolic  evaluation  The  system  is 
written  in  PU1  and  runs  on  an  IBM 


370(168  Therein  a  command  language 
available  for  selective  execution  of 
parts  of  SADAT. 

SADAT  appears  to  be  a  wall  engi¬ 
neered.  habitable  testing  facility,  with 
different  tools  integrated  ia  a  com¬ 
plementary  manner 

L  Clarke  presented  the  work  of  bar 
group  at  the  University  of  Masae- 
chuaetts  (co-workers  are  Neal  Ogden 
and  Daryl  Winters)  in  the  design  of  a 
system  called  Attest,  to  be  used  for 
symbolic  execution  of  Fortran  pro¬ 
grams  in  the  context  of  top-down 
tasting.  The  Attest  interface  deecnp- 
tion  language  A I D  enables  the  user  to 
describe  both  predicated  and  pre¬ 
sumed  relationships  among  program 
variables.  Thu  feature  is  important  in 
top-down  testing  lor  more  generally,  in 
testing  using  stubs  for  modules  not 
yet  written),  since  the  specifications  of 
the  modules  can  be  stated  by  means  of 
AID  commands,  and  symbolic  execu¬ 
tion  can  proceed  as  if  the  module  ia 
written  and  connected  AID  has  condi¬ 
tional  execution  constructs  for  the 
easy  description  of  conditional  pro¬ 
cedure  computations  in  early  versions 
of  a  program.  Fortran  and  AID  can  be 
freely  mixed  so  that  a  module  can  be 
executed  normally  or  symbolically. 
Attest  also  supports  symbolic  I/O.  Us¬ 
ing  these  features,  the  developer  can 
produce  successive  refinements  with 
progressively  less  AID  and  more  For 
Iran  code.  For  the  class  of  applications 
where  symbolic  execution  is  useful, 
the  Attest  system  can  bring  program 
creation  and  testing  closer  together 
and  help  realize  the  promise  of  step¬ 
wise  refinement. 

R.  N.  Taylor  of  Boeing  Computer 
Services  and  L.  J  Osterweil  of  the 
University  of  Colorado  reported  on 
their  work  in  developing  static  and 
dynamic  testing  techniques  for 
concurrent-process  programs.  This 
work  was  performed  in  connection 
arith  N  ASA-Langley  s  Must  program, 
which  addresses  the  production  and 
testing  of  concurrent-process  flight 
software. 

Dynamic  testing  of  single-process 
programs  often  includes  generation  of 
histograms.  These  describe  a  pro¬ 
gram  s  execution  history  by  display¬ 
ing  counts  of  statement  and  branch 
point  executions.  Taylor  and  Osier 
weil  proposed  the  notion  of  a  pro refe¬ 
rrals  Histogram  as  an  extension  of  this 
technique  for  concurrent-process  pro¬ 
grams.  Each  lime  an  event  change 
Ukes  place,  a  process  slate  snapshot 
is  made  indicating  the  state  of  dif¬ 
ferent  processes.  A  series  of  such 
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snapshots  is  used  to  compuU  the  pro¬ 
cess  state  histogram.  Automatic 
monitoring  of  system  deadlock  errors, 
which  can  occur  in  concurrent-process 
programs,  can  be  incorporated  by  us¬ 
ing  several  available  algorithms. 
Dynamic  assertion  verification  can  be 
extended  to  concurrent-process  soft¬ 
ware  and  is  especially  valuable  to 
assure  that  scheduling  and  timing 
constraints  are  as  designed 

Static  analysis  is  often  effective  in 
weeding  out  errors  that  are  coetlier  to 
detect  by  dynamic  testing  techniques. 
Extension  of  data  flow  analysis  to 
concurrent-process  software  requires 
more  complex  control  flow  models. 
The  PAF— process  augmented  flow- 
graph— is  a  concept  designed  to  cap¬ 
ture  the  data  and  control  flows  in 
concurrent-process  programs  with 
tehrdult  and  wait  statements  as  syn¬ 
chronization  constructs.  The  PAF  and 
associated  algorithms  are  capable  oi 
detecting  errors  due  to  shared  data 
items  being  referenced  by  one  process 
before  any  other  process  defines  them. 
In  addition,  certain  anomalies  in  the 
PAF  indicate  the  occurrence  of  poorly 
coordinated  processes.  While  PAFs 
are  useful  for  a  class  of  concurrent  con¬ 
structs.  further  work  remains  to  be 
done  for  a  broader  class  of  syn¬ 
chronization  constructs,  such  as  open, 
close,  and  ugnal  statements. 

M.  H  ok  house  and  M.  Hatch  of 
Analytic  Sciences  Corporation  dis¬ 
cussed  their  experience  with  a  set  of 
tools  including  ones  for  static 
analysis,  assertion  processing,  and 
test  data  generation  for  a  path 
coverage  based  approach  While  their 
experience  indicated  substantial 
benefits  from  the  interactive  use  of 
these  tools,  they  also  discovered  some 
potential  problems  Sometimes  pro¬ 
tection  schemes  are  devised  to  make 
each  module  "robust "  against  its  en 
vironment.  During  integration,  the 
protection  schemes  of  two  modules 
may  overlap,  causing  some  protection 
branches  in  the  low-level  module  to 
become  unreachable.  Another  issue  is 
large  system  testing.  Each  module  can 
be  tested  separately  for  high  coverage, 
but  when  they  are  integrated  discon¬ 
tinuities  in  overall  system  flow  may  be 
difficult  to  detect.  For  loop  test¬ 
ing  Holthouse  and  Hatch  suggested 
that  each  loop  be  tested  not  only  for  its 
looping  state,  but  also  for  a  program 
flow  not  passing  through  the  loop  at 
all.  While  the  testing  tools  they 
discussed  cannot  detect  missing 
paths.  Holthouse  and  Hatch  pointed 
out  that  the  close  software  inspection 


the  approach  forced  them  to  make  did 
in  fact  lead  to  the  discovery  of  several 
errors  of  this  type. 

One  of  the  accomplishments  of  the 
workshop  was  the  establishment  of  a 
mechanism  for  the  exchange  of  infor¬ 
mation  on  implemented  test  tools.  Dr. 
Selden  Stewart  of  the  National  Bureau 
of  Standards  has  consented  to  be  the 
coordinator  of  this  activity.  If  you  are 
interested,  contact  him  at  the  Na¬ 
tional  Bureau  of  Standards.  Tech 
A-265.  Washington.  DC  20234,  13011 
921-3485.  He  will  be  preparing  a  ques¬ 
tionnaire  to  obtain  information  about 
available  tools,  their  language  and 
machine  constraints,  documentation, 
and  conditions  of  release. 

With  respect  to  future  work  in  tool 
development,  there  is  a  real  need  for 
systematic  evaluation  of  tools  both  in 
the  context  of  testing  programs  in 
practical  environments  and  in  the  con¬ 
text  of  carefully  controlled  experimen¬ 
tal  situations  This  will  yield  insights 
about  the  relationship  between  error 
type*.  tools,  and  types  of  tasks  Some 
other  aspects  requiring  attention  in 
elude  development  of  tools  for  a  larger 
class  of  concurrent-process  programs, 
incorporation  of  tools  into  the  average 
programmer  s  environment,  and 
human  factors  in  tool  design  The  last 
aspect  is  important  because  it  i» 
unlikely  that  software  testing  will 
soon  become  entirely  automated  The 
human  tester  will  continue  to  plav  a 
decisive  interactive  role 
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Test  date  generation:  three 
approachea  prevail 

D  W.  Fife 

National  Bureau  of  Standards 

Test  data  generation  could  be  de¬ 
fined  simply  as  a  collection  of  tech¬ 
niques  for  creating  valid  input  data, 
considered  in  terms  of  feasibility, 
economy,  and  efficiency.  But  it  is  more 
important  to  discuss  lest  data  genera¬ 
tion  in  the  overall  context  of  testing 
practice.  Testing  practice  includes  the 
testing  methodology-  and  tools  that 
give  the  basis  for  test  data  generation 
1 1  also  involves  many  other  issues  that 
affect  evaluation  and  acceptance  of 
dau  generation  methods,  such  as 
testing  administration,  documenta¬ 


tion.  and  auditing.  For  example,  it  ia 
essential  to  be  able  to  distinguish  test 
cases  and  to  accurately  describe  them 
relative  to  program  specifications  and 
testing  goals.  Test  cases  must  be 
repeatable,  particularly  for  real-time 
systems.  Testing  must  be  auditable 
and  documented  so  that  the  noftwsre 
user  or  customer  is  assured  that  the 
proper  tests  have  been  applied  and  in 
the  proper  manner. 

For  the  purpose  of  discussing  test 
data  generauon.  we  can  define  testing 
as  the  execution  of  a  program  on  finite 
input  in  order  to  infer  conformance  to 
specifications  Specifications  agreed 
upon  by  user  and  developer  are  usual¬ 
ly  incomplete  or  else  lack  rigor  and 
precision  in  many  aspects  The  user  or 
the  tester  may  need  to  augment  or 
refine  the  specification  so  that  the 
result  of  any  lest  is  precisely  deter¬ 
mined.  The  tester  s  problem  then  is  to 
use  i possibly  augmented)  specifica¬ 
tions  along  with  accepted  testing  prin¬ 
ciples  and  his  own  knowledge  to  derive 
appropriate  test  cases. 

Many  workshop  presentations 
touched  on  t*  A  dau  generation  prob¬ 
lems  For  example,  the  research  on 
program  muution  presented  by  Fred 
Sayward  of  Yale  and  others  aims  to 
evaluate  a  programmer's  selected  test 
cases,  but  does  not  define  or  prescribe 
how  the  test  dau  is  produced.  Many 
tools  such  as  SADAT  (developed  by 
Voges.  Amschter.  and  Gmeiner  of 
Karlsruhe.  West  Germanyl  include 
computer  analysis  of  program  predi¬ 
cates  to  help  the  programmer  select 
lest  dau  to  exercise  all  logical  paths. 
But  solutions  of  the  predicates  that 
would  specify  the  needed  data  are  not 
readily  obtainable  land  arc  undecida- 
ble  in  the  general  case),  so  the  pro¬ 
grammer  must  make  his  own  analyais 
and  frequently  mutt  choose  test  dau 
by  tnal  and  error. 

The  software  testing  field  still  lacks 
one  technique,  or  a  set  of  them,  that 
would  be  widely  accepted  as  suffi¬ 
ciently  effective  to  be  a  conclusive 
testing  method.  It  is  commonly  be¬ 
lieved  that  a  minimum  requirement  ia 
to  test  every  program  sutement  and 
branch  at  least  once.  This  criterion  is 
often  marginal1 1  but  can  be  sup¬ 
plemented  by  other  criteria  aa 
workshop  participant  Mark  Holt¬ 
house  of  Analytic  Sciences  Corpora¬ 
tion  described.  As  a  propoaed 
minimum  testing  sundard.  this  war 
rants  a  more  extensive  empirical 
evaluation  Sometimes,  it  leads  to 
many  more  test  cases  than  may  be 
economically  accepuble.  Also,  it  is 
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